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FOREWORD 


The  purpose  of  this  study,  which  was  conducted  in  support  of  the  Office 
of  Naval  Research's  Operational  Decision  Aids  Project,  was  to  investigate 
empirically  the  significance  of  perceived  risk  considerations  in  designing 
displays  for  Navy  tactical  decision  making. 
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are  gratefully  acknowledged.  Mr.  Ramon  Hershman  also  made  valuable  con- 
tributions to  the  project  as  software  and  systems  analyst. 
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SUMMARY 


Problem 

The  Office  of  Naval  Research,  as  part  of  a large  project  in  Operational 
Decision  Aiding,  has  contracted  for  the  development  of  several  decision 
analytic  tools  to  aid  in  tactical  decision  making.  Much  of  this  development 
has  occurred  despite  a lack  of  sufficient  knowledge  about  the  importance  of 
perceived  risk  considerations  in  Navy  operational  decision  making.  "Risk" 
is  defined  as  the  relative  certainty  of  the  outcome  associated  with  a 
selected  action.  If  an  individual  is  confident  of  the  outcome  that  will 
result  from  a particular  action  on  his  part,  the  perceived  risk  of  selecting 
that  act  is  said  to  be  low.  Factors  of  risk  have  been  observed  to  be  very 
important  in  many  decision  situations  and  may  be  an  important  presence  here 
as  well.  If  so,  risk  may  be  too  potent  a factor  to  disregard  in  the  develop- 
ment of  these  decision  aids. 

Objective 

The  present  study  is  an  empirical  investigation  of  the  importance  of 
perceived  risk  considerations  in  Navy  operational  decision  making. 

Approach 

Decision  performance  by  12  expert  subjects  using  two  versions  of  a 
decision  aiding  approach  were  compared.  One  version  used  a prototype 
decision  aiding  display  that  was  developed  by  Decisions  and  Designs,  Inc.  (DDI) . 
This  display  contained  a minimal  amount  of  information  about  the  risk 
involved  in  the  selection  of  the  action  alternatives  available  to  the 
decision  maker.  The  other  version  used  a display  which  explicitly  pre- 
sented some  aspects  of  risk  and  which  included  the  same  type  of  information 
that  was  present  in  the  display  by  Decisions  and  Designs,  Inc.  Included  in 
the  display  was  information  indicating  the  range  of  outcomes  associated 
with  each  action  and  the  probability  that  each  of  these  outcomes  would  occur. 

The  subjects  used  the  displays  to  aid  them  in  analyzing  realistic  tactical 
scenarios  that  were  presented  to  them  under  supposed  conditions  of  high  and 
low  risk. 

Results  and  Conclusions 

The  degree  to  which  the  decision  aids  improved  decision  performance 
appears  to  be  dependent  upon  the  individual's  response  toward  risk.  Use 
of  the  DDI  display  was  significantly  beneficial  to  those  individuals  who 
demonstrated  a greater  aversion  to  risk  taking  when  placed  in  a high  risk 
situation.  When  using  the  risk  explicit  display,  these  risk  averse  indi- 
viduals showed  no  significant  performance  imporvement  over  the  no-aid  con- 
dition. Subjects  who  became  less  conservative  or  less  risk  averse  under 
a high  risk  situation  did  not  benefit  by  the  use  of  these  aids. 

Although  there  was  no  evidence  to  support  the  contention  that  explicit 
risk  information  would  enhance  performance,  the  subjects  in  the  investiga- 
tion felt  that  risk  information  could  very  possibly  be  more  important  in 
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other  tactical  settings  than  were  explored  in  this  study.  Thus,  in  the 
future  development  of  decision  analytic  aids  further  consideration  should 
be  given  the  advisability  of  providing  access  to  risk  information  upon 
request. 

Additionally,  the  aspects  of  risk  utilized  in  this  investigation  were 
but  a subset  of  the  possible  dimensions.  The  failure  of  maximum  subjective 
utility  to  completely  account  for  the  subjects'  action  selections  suggests 
that  other  characteristics  of  the  decision  are  operating  and  may  very  well 
represent  some  dimensions  of  risk  which  were  not  explored  here. 

Recommendations 

1.  Continue  development  of  a maximum  subjective  expected  utility  (SEU) 
approach  for  decision  aiding  as  in  the  Decisions  and  Designs  prototype. 

2.  Identify  decision  making  styles  and  the  type  of  decision  aiding 
approaches  that  benefit  and  hinder  each. 

3.  Consider  the  use  of  group  estimation  procedures  to  obtain  likelihood 
ratio  estimates  of  increased  reliability. 

4.  In  a Bayesian  approach  such  as  the  DDI  prototype,  the  user  of  the 
decision  aid  should  have  the  ability  to  update  or  change  likelihood  ratios 
to  reflect  conditional  dependencies  in  the  data. 

5.  The  decision  task  in  the  DDI  prototype  should  include  an  assessment 
of  the  probability  that  the  enemy  has  changed  its  intent  instead  of  assuming 
that  all  the  data  observed  came  from  a steady  generating  state. 
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INTRODUCTION 


Problem 


Decisions  and  Designs,  Inc.  (DDI) , an  Office  of  Naval  Research  (ONR) 
contractor,  has  developed  a prototype  decision  analytic  tool  for  use  in 
the  ONR  Operational  Decision  Aids  Project.  In  decision  making,  it  is 
usually  recommended  that  the  decision  maker  (Dll)  select  that  alternative 
that  will  maximize  his  subjective  expected  utility  (SEU)  or  outcome. 
Accordingly,  the  DDI  decision  aid  displays  the  result  of  a decision  analysis 
as  a probability  triangle  (Brown,  Peterson,  Shawcross,  & Ulvila,  1975), 
which  continuously  informs  the  DU  as  to  the  current  probabilities  of  the 
states  of  the  world  (enemy  intent)  and  indicates  a recommended  act  which 
will  maximize  SEU  given  these  probabilities.  However,  action  recommenda- 
tions based  solely  on  this  criterion  may  not  always  be  appropriate  for  a 
Navy  DM.  Given  the  choice  between  two  acts  with  approximately  equal  SEU, 
he  might  well  prefer  the  act  with  the  least  perceived  risk;  that  is,  one 
in  which  he  is  somewhat  more  confident  of  the  resultant  outcome.  He  might 
even  be  willing  to  select  an  act  that  has  lower  SEU  if  it  also  involves 
lower  risk.  This  risk  is  reflected  in  the  range  or  variance  of  the  possible 
outcomes  as  opposed  to  the  mean  of  the  outcomes  as  given  by  the  SEU. 


The  DDI  prototype  does  not  provide  for  the  perceived  risk  which  may  be 
associated  with  the  various  alternatives. 


Background 


The  importance  of  risk  considerations  in  addition  to  expected  utility 
has  been  addressed  in  several  studies  (Allais,  1953;  Coombs  & Pruitt,  1960; 
Slovic,  1967;  Slovic  & Lichtenstein,  1968;  Payne  & Braunstein,  1971;  Payne, 
1975).  Unfortunately,  the  relationships  between  risk  and  alternative  pre- 
ferences have  been  shown  to  be  somewhat  complicated.  In  fact,  few  attempts 
have  been  made  to  determine  the  nature  of  risk,  the  most  formal  being  that 
by  Pollatsek  and  Tversky  (1970).  However,  these  studies  have  suggested 
the  properties  which  seem  to  reflect  the  risk  associated  with  each  act,  such 
as  variance,  probabilities  of  the  outcomes,  maximum  loss  or  regret,  the 
range  of  outcomes,  or  some  combination  of  these  parameters  (Rapoport  & 
Wallsten,  1972). 


The  variance  of  the  utilities  associated  with  an  act  is  a measure  of 
the  dispersion  of  the  utilities  for  the  outcomes  that  may  result  as  a 
consequence  of  the  act.  Low  variance  for  an  act  indicates  that  the  utility 
for  any  outcome  resulting  from  the  act  will  be  relatively  close  in  value  to 
its  SEU;  and  high  variance,  that  it  may  be  quite  different,  relatively, 
from  its  SEU.  Thus,  different  attitudes  toward  risk  may  possibly  be  based 
on  preferences  for  variability  of  the  utility  of  outcomes.  By  selecting  a 
low  variance  act,  a DU  can  be  relatively  confident  that  actual  utility  for 
the  act  will  be  close  to  its  SEU. 
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Risk  considerations  can  be  incorporated  within  the  SEU  maximization 
framework  by  using  a "tv-BR.LT"  technique  (Raiffa,  1968;  Barclay,  Brown, 
Chinnis,  Kelly,  & Peterson,  1973)  which  is  a method  for  estimating  an 
individual's  utility  for  an  outcome.  The  individual  is  asked  to  adjust 
the  probability  values  of  a basic  reference  lottery  such  that  he  would 
be  indifferent  to  participating  in  the  lottery  or  receiving  the  outcome  for 
certain.  Since  the  indifference  point,  as  established  in  the  context  of  a 
gamble,  is  that  point  where  the  individual  could  possibly  obtain  both  more 
or  less  than  the  amount  offered  for  certain,  the  expected  value  of  the 
adjusted  lottery  is  viewed  as  the  risk-incorporating  utility  for  the  certain 
outcome.  However,  recent  research  (Slovic,  1975;  Tversky,  1972)  suggests 
that  the  results  of  procedures  such  as  the  "ir-BRLT"  method  do  not  necessarily 
generalize  to  choice  situations  in  a satisfactory  manner. 

Since  little  is  known  about  the  importance  of  perceived  risk  considera- 
tions in  Navy  operational  decision  making,  the  Navy  Personnel  Research  and 
Development  Center  proposed  an  empirical  study,  utilizing  the  DDI  prototype 
and  a second  decision  aid  developed  by  NAVPERSRANDCEN  that  incorporated 
possible  risk  factors,  to  investigate  the  significance  of  such  factors 
in  tactical  decision  making  at  the  task  force  commander  (TFC)  level.  The 
basic  rationale  for  the  study  was  not  to  examine  risk  per  se  but,  rather, 
to  determine  if  risk  is  a potent  enough  factor  to  warrant  attention  in 
the  ONR  decision  aids. 


Obi ective 


As  indicated,  the  objective  of  this  study  was  to  investigate  the 
significance  of  perceived  risk  in  decision  making  by  a Navy  TFC.  To  achieve 
this  end,  it  was  necessary  (1)  to  implement,  via  computer,  the  DDI  and 
NAVPERSRANDCEN  decision  aids,  (2)  to  develop  realistic  tactical  scenarios 
as  a setting  within  which  to  compare  the  two  aids,  and  (3)  to  secure  subjects 
having  detailed  knowledge  about  Soviet  tactical  activity  and  experience 
in  evaluating  such  activity. 


To  determine  whether  the  availability  of  various  "ingredients"  of  the 
SEU  equation,  in  addition  to  those  provided  by  the  DDI  prototype,  would 
improve  or  degrade  the  performance  of  the  DM,  the  Center  studied  the  effect 
of  the  two  displays  under  different  conditions  of  perceived  risk.  Perceived 
risk  was  manipulated  implicitly  by  varying  the  risk  dimensions  while  main- 
taining the  same  type  of  decision  scenario.  Obviously,  if  risk  considera- 
tions are  found  to  be  a major  factor  in  naval  decision  making,  it  would  be 
desirable  to  use  a display  that  presents  the  necessary  elements  for  the  DM 
to  assess  the  risk  involved. 
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METHOD 


This  study  compared  the  decision-aiding  capabilities  of  two  displays — 
the  DDI  triangle  (risk  implicit)  and  the  Center  developed  bar  graph  (risk 
explicit) — in  tactical  scenarios  under  prevailing  conditions  of  high  or 
low  risk.  During  a scenario,  a decision  maker  (DM)  could  either  refer  to 
one  of  the  displays  or,  in  a control  condition,  to  no  aiding  display.  As 
data  events  occurred  sequentially,  the  DM  classified  each  event,  entered 
this  classification  to  the  decision  aid,  observed  the  resultant  integration 
of  the  entry,  appraised  enemy  intent,  and  recommended  an  act  in  response 
on  the  basis  of  information  available.  The  DM  used  either  one  of  the 
decision  aiding  displays  or  no  aid  during  different  blocks  of  the  experiment, 
with  each  DM  serving  as  his  own  control  and  with  the  various  conditions 
counterbalanced.  Each  scenario  presented  to  the  DM  was  designated  as  either 
a high  or  low  risk  condition. 

Subjects 

The  Center  obtained  formal  consent  from  the  Chief  of  Naval  Education 
and  Training  (CNET)  for  the  cooperation  of  the  Fleet  Combat  Direction  Systems 
Training  Center,  Pacific  (FCDSTCP)  in  this  project.  FCDSTCP  is  the  West 
Coast  site  of  the  Navy's  Tactical  Action  Officer  (TAO)  school,  which  trains 
officers  in  analysis  of  and  response  to  enemy  tactical  activity.  Twelve 
FCDSTCP  naval  officers  who  serve  as  TAO  course  instructors  participated  as 
subj  ects . 

Scenarios 


Scenarios  developed  were  based  on  the  ONRODA  warfare  scenario  (Payne  & 
Rowney,  1975),  and  amplified  by  appropriate  risk-manipulating  scene  setters. 

To  ensure  these  scenarios  were  based  upon  up-to-date  intelligence  about 
enemy  weapons  and  tactical  employment,  the  third  author  attended  a 1-week 
Multi-Threat  Overview  course  conducted  by  FCDSTCP . Also,  current  Navy 
intelligence  publications  were  extensively  researched  and  consultations  were 
held  with  FCDSTCP  staff  personnel.  As  a result  18  realistic  tactical  scenarios 
were  constructed,  nine  of  which  were  actually  used  in  the  experiment.  The 
scenarios  unfold  in  an  open-ocean  environment  and  involve  transit  of  a U.S. 
task  force  (the  Blue  force),  composed  of  a single  aircraft  carrier  CV  and 
three  destroyer  escorts,  toward  ONRODA  Island.  The  mission  of  this  task 
force  is  to  arrive  in  the  ONRODA  area  to  provide  a continuous  capability 
for  projecting  strikes  onto  ONRODA  Island  and  the  Greyhawk-held  areas  in 
Grey  (Payne  & Rowney,  1975).  An  integral  part  of  this  mission  is  to  conserve 
task  force  assets  as  much  as  operationally  feasible  during  the  transit 
so  as  to  arrive  in  the  ONRODA  area  with  maximum  strike  capability.  However, 
during  its  move  to  station,  the  task  force  is  subject  to  a full  spectrum 
of  possible  Red  tactical  activity — ranging  from  routine  surveillance  activity 
to  harassment  of  the  U.S.  force  to  actual  attack  on  the  task  force.  These 
various  Red  intents  are  defined  as  follows: 
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1.  ROUTINE — Red  intends  to  engage  in  only  his  routine  (as  understood 
by  the  task  force  commander)  surveillance  and  training  activities  during 
U.S.  task  force  transit.  There  is  no  intent  to  prevent,  by  harassment, 
the  U.S.  force  from  carrying  out  its  mission  nor  any  intent  to  attack  the 
U.S.  task  force. 

2.  HARASS — Red  intends,  in  some  manner  short  of  actual  attack,  to 
prevent  the  U.S.  task  force  from  carrying  out  its  assigned  mission  in  a 
timely  manner  or  intends  to  cause  expenditure  of  task  force  assets  above 
routine  level. 

3.  ATTACK — Red  intends  to  attack  the  U.S.  task  force. 

As  the  scenario  unfolded,  the  DM  was  required  to  decide  the  intent  of 
each  Red  activity,  as  defined  above,  and  then  decide  upon  an  appropriate 
response  by  his  own  force,  weighed  against  the  risk  of  degrading  the  force's 
ability  to  eventually  carry  out  its  original  mission,  llis  three  choices 
of  action  are  defined  as  follows: 

1.  ROUTINE — Maintain  only  that  readiness  posture  and  expend  only  those 
task  force  assets  during  the  transit  appropriate  for  a Red  intent  of  routine 
operations  so  as  to  arrive  in  the  OHRODA  area  on  time  and  with  maximum  assets 
available. 

2.  STAGE — Operate  at  a readiness  posture  and/or  expend  task  force  assets 
at  some  level  above  routine  but  short  of  attack. 

3.  ATTACK — Attack  the  Red  forces. 

The  rules  of  engagement  (ROE)  governing  the  task  force  commander's 
actions  in  the  scenarios  were  as  shown  below: 


WHEN  CERTAIN3  THAT 
RED  INTENT  IS 


TAKE  TASK 
FORCE  ACTION 


ROUTINE 


ROUTINE 


HARASS 


STAGE 


ATTACK 


ATTACK^ 


aIn  cases  less  than  certainty,  take  that  action  you  judge  to  be  most 
appropriate. 

^You  may  shoot  first  if  you  feel  you  have  strong  enough  indication  that 
Red  intends  to  attack. 

Risk  Conditions 

Scenarios  were  presented  against  either  a high-risk  or  a low-risk 
background  for  decision  making. 


For  the  high-risk  background,  referred  to  as  the  "one  CV"  case, 
subjects  were  told  that  the  task  force  for  which  they  were  functioning  as 
TFC  was  the  only  one  available  to  carry  out  the  ONRODA  mission.  Therefore, 
if  it  failed  to  reach  the  ONRODA  area  on  schedule  or  reached  there  with 
degraded  assets,  the  mission  could  not  be  fully  accomplished. 


For  the  low-risk  background,  referred  to  as  the  "three  CV"  case, 
subjects  were  told  to  assume  that  there  were  two  additional  U.S.  task  forces 
enroute  to  ONRODA.  Although  the  geographical  location  of  these  task  forces 
precluded  the  DM  from  interacting  with  them  during  the  transit,  there  was 
a possibility  that,  upon  their  arrival  in  the  ONRODA  area,  they  could  assume 
some  or  all  of  the  mission  assigned  to  the  subject's  own  task  force  if  it 
failed  to  arrive  on  schedule  or  arrived  with  degraded  assets. 


Decision  Aiding  Conditions 


DPI  Triangle 


The  DDI  decision  aid  is  described  in  detail  in  Brown,  Peterson, 
Shawcross,  and  Ulvila  (1975).  Rather  than  reproduce  the  aid  in  its  entirety 
for  this  study,  NAVPERSRANDCEN  incorporated  only  those  features  that  were 
necessary  to  study  risk  effects.  Utilization  of  the  NAVPERSRANDCEN  version 
of  the  DDI  aid  required  classification  of  a scenario  data  event  in  terms  of 
its  threat  to  the  task  force.  The  likelihoods  associated  with  the  designated 
level  of  threat  were  used  by  the  decision  aid  to  perform  a Bayesian  revision 
of  the  prior  probabilities  that  each  of  three  hypotheses  was  true.  A 
decision  "bug"  moved  about  in  a triangular  probability  space  according  to  the 
posterior  probabilities  computed  with  each  data  indicator  designated  to  have 
occurred.  The  triangular  probability  space  was  delineated  into  "action  areas,' 
which  were  determined  by  a utility  matrix  appropriate  for  the  decision  to  be 
made.  This  matrix  reflected  the  DM's  valuation  for  the  resultant  outcomes 
of  his  action  alternatives,  based  upon  the  condition  of  each  hypothesized 
state  of  the  world  being  true.  Based  upon  the  utility  matrix,  the  action 
areas  simply  represented  those  locations  in  the  triangular  probability  space 
for  which  a particular  action  would  have  the  greatest  SEU.  Thus,  the  loca- 
tion of  the  "bug"  in  any  action  area  identified  that  action  as  the  aid's 
recommendation,  based  upon  the  DM's  utilities  and  the  data  events  that  had 
occurred.  Illustrations  of  sequences  of  data  events  and  how  they  were  dis- 
played in  the  Center  version  of  the  DDI  aid  is  given  in  Figures  1 through  3. 

In  these  figures,  marks  the  location  of  the  initial  prior  probabilities, 

"N"  represents  the  newest  set  of  posterior  probabilities,  and  "0"  represents 
the  second  most  recent  set  of  posterior  probabilities.  Other  items  appearing 
in  these  figures  are  described  in  the  "Procedure"  section. 


SCENARIO  1 Page 


6 


Figure  1.  DDI  Triangle  Display  (Trial  1) . 
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DDI  Triangle  Display 


Triangle  Display  (Trial  3) 
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NAVPLRSRANDCEN  Bar  Graph 

The  basic  information  contained  in  the  DDI  triangle  was  provided 
by  the  NAVPERSRANDCEN  bar  graph,  plus  some  additional  elements.  Host 
importantly,  the  graphical  display  was  designed  to  more  directly  reflect 
utilities  and  probabilities  of  the  various  possible  outcomes.  For  each 
outcome,  utilities  were  displayed  explicitly  as  a bar  located  along  a 
utility  scale  provided  for  each  action  alternative.  The  heights  of  the 
bars  reflected  the  posterior  probabilities  of  the  outcomes  associated 
with  each  act.  By  showing  the  actual  utilities  and  probabilities  of 
outcomes  in  the  display,  it  was  possible  to  provide  risk-related  informa- 
tion explicitly  at  each  decision  opportunity.  To  ensure  that  the  DM 
had  the  same  information  that  he  had  available  in  the  DDI  display,  a 
small  triangular  indicator  was  placed  along  the  utility  scale  for  each 
act  to  indicate  the  SEU  associated  with  that  act.  The  DM,  therefore, 
could  make  a decision  based  not  only  upon  a criterion  of  SEU  maximization 
but  also  upon  the  relative  utilities  and  probabilities  of  the  outcomes 
within  each  act  (thereby  providing  the  DM  the  information  to  assess  risk 
if  he  chose).  Figures  4 through  6 illustrate  the  NAVPERSRANDCEN  bar  graph 
display  for  the  same  data  events  that  were  used  in  illustrating  implementa- 
tion of  the  DDI  decision  aid. 

V.o  Aid  Condition 

During  the  control  condition,  the  DM  had  to  perform  the  same  data 
classification  tasks  that  were  required  using  one  of  the  aids  described 
above  but  with  no  decision-aiding  device  available. 

Des ign 

The  study  basically  consisted  of  six  possible  conditions  of  aiding 
combinations. 


ii 

1.  DDI  triangle — high  risk. 

2.  DDI  triangle — low  risk. 

3.  NAVPERSRANDCEN  bar  graph— high  risk. 

4.  NAVPERSRANDCEN  bar  graph — low  risk. 

5.  No  aid — high  risk. 

. 

6.  No  aid — low  risk. 

Although  each  subject  participated  in  only  three  of  the  six  combinations, 
he  did  participate  in  all  three  aiding  conditions  and  in  both  risk  conditions 
at  least  once.  The  order  in  which  subjects  participated  in  the  various 
conditions  was  counterbalanced  over  subjects.  The  assignment  of  subjects 
to  conditions  is  shown  in  Table  1. 
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Figure  5.  NAVPERSRANDCEN  Bar  Graph  Display  (Trial  2). 
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Figure  6.  NAVPERSRANDCEN  Bar  Graph  Display  (Trial  3). 


Table  1 


Experimental  Design 


Subject 

DDI 

Triangle 

NAVPERSRANDCEN 
Bar  Graph 

No 

Aid 

High 

Risk  Low  Risk 

High  Risk  Low  Risk 

High  Risk  Low  Risk 

Si 

X 

X 

X 

s2 

X 

X 

X 

S3 

X 

X 

X 

s4 

X 

X 

X 

S5 

X 

X 

X 

S6 

X 

X 

X 

S 7 

X 

X 

X 

S8 

X 

X 

X 

s9 

X 

X 

X 

S10 

X 

X 

X 

Six 

X 

X 

X 

S 1 2 

X 

X 

X 

Two  scenarios  of  different  types  (routine,  stage,  or  attack)  were 
presented  to  the  subject  within  each  experimental  condition  in  which  he 
participated.  Thus,  a subject  was  presented  with  six  different  scenarios. 

The  scenarios  were  counterbalanced  for  type  so  that  the  subject  saw  each 
type  exactly  twice — but  never  twice  in  the  same  condition. 

Following  the  three  experimental  conditions,  a fourth  condition  was 
administered  which  was  matched  to  the  condition  in  which  the  subject 
was  given  the  DDI  triangle.  For  example,  if  the  subject  originally  saw 
the  DDI  triangle  under  the  "high  risk"  condition,  he  was  again  presented 
with  the  DDI  triangle  under  the  high  risk  condition.  However,  in  the  extra 
condition,  the  boundaries  of  the  action  areas  in  the  triangle  were  determined 
by  the  risk-incorporating  utility  matrices,  which  were,  in  turn,  estimated 
by  a method  of  theoretical  extrapolation  discussed  below.  The  scenarios 
presented  during  this  additional  condition,  although  different  from  the 
first  six,  were  also  selected  to  match  the  enemy  intent  when  the  subject 
originally  saw  the  DDI  triangle. 

Stimulus  Material 

The  computer  facility  used  in  this  study  was  not  cleared  to  utilize 
classified  information.  Thus,  the  scenarios,  which  were  based  upon  a great 
deal  of  SECRET  material  could  not  be  presented  by  a computer-generated  dis- 
play but  were  confined  to  printed  SECRET  copy.  Each  data  occurrence  was 
presented  on  a separate  page  that  contained  (1)  printed  narrative  along 
with  its  concomitant  status  board  information  and  (2)  a group  of  classifica- 
tions, one  of  which  was  to  be  chosen  by  the  DM  as  that  most  descriptive  of 
the  data  event  presented. 

Each  scenario  consisted  of  20  such  events  combined  to  form  a small 
folder,  the  pages  of  which  are  successively  examined  as  the  DM  progressed 
through  the  scenario.  At  any  page  in  the  scenario,  the  status  boards 
contained  on  that  page  showed  a summary  of  the  data  events  occurring  up 
until  that  point.  Appendix  A provides  an  example  of  a scenario,  purged 
of  classified  material,  used  in  the  study. 

Questionnaire 

A short  questionnaire  (Appendix  B)  was  administered  to  each  subject 
after  he  completed  the  experimental  sessions.  The  questionnaire  consisted 
of  several  items  to  which  the  subject  gave  responses  along  a five-point 
Likert  scale.  Some  items  were  intended  to  measure  the  individual's 
attitude  toward  using  computer-based  decision  aids,  in  general,  and  the 
decision  aids  presented  during  the  experiment,  in  particular;  and  others, 
to  assess  the  adequacy  of  the  indoctrination  briefing  and  to  check  on  the 
attitudinal  effect  of  the  risk  manipulation.  Additional  information  was 
gathered  to  provide  an  experience  profile  of  the  subject  population. 


Apparatus 

The  two  decision  aids  were  implemented  on  the  Center's  Digital  Equip- 
ment Corporation  PDP-12  computer  and  interfaced  Tektronix  4006-1  graphics 
terminal  using  a locally  modified  version  of  the  SP-12  programming  language. 

A subject  used  the  terminal  to  input  information  into  and  to  receive 
information  from  the  aid.  The  computer  was  also  used  to  systematically 
query  the  DII  and  to  record  information  during  a scenario's  evolvement  and 
the  progression  of  the  DM  through  the  scenario. 

Procedure 

A project  briefing,  lasting  from  3 to  3 1/2  hours,  was  given  to  sub- 
jects, in  groups  of  three  at  a time.  Following  a general  description  of 
the  ONR  decision-aiding  project  and  objectives,  it  was  stressed  that  aids 
to  be  utilized  by  subjects  in  this  experiment  were  prototypic  in  form  and 
constituted  only  a small  portion  of  the  entire  ONR  decision-aiding  package. 

The  briefing  continued  with  a general  introduction  to  the  ONRODA  scenario 
and  a description  of  the  subject's  role  as  commander  of  a U.S.  task  force 
enroute  to  the  ONRODA  area.  Differences  between  the  one  CV  and  three  CV 
cases  were  detailed.  Potential  enemy  activity  during  the  transit  was 
explained  to  come  only  from  Red,  who  could  be  taken  to  be  the  Soviets. 

The  requirement  for  the  TFC  to  determine  both  Red  intent  and  an  appropriate 
Blue  task  force  response,  defined  as  in  the  "Scenarios"  section  above,  was 
described . 

The  rules  of  engagement  included  a provision  for  the  Blue  force  to 
shoot  first,  which  is  exactly  opposite  to  rules  under  which  our  subjects 
have  operated  in  real  world  tactical  situations.  Thus,  to  justify  this 
action,  subjects  were  told  to  assume  that,  in  the  recent  past,  a U.S. 
destroyer  on  an  intelligence-gathering  mission  off  the  coast  of  a small 
Red  ally  was  hit  and  sunk  by  an  antiship  cruise  missile.  This  sinking 
occurred  in  spite  of  numerous  electronic  indications  aboard  the  destroyer 
that  a missile  attack  was  imminent.  The  Navy,  in  response  to  the  resulting 
Congressional  criticism,  changed  the  old  "don't  fire  unless  fired  upon" 
rule  to  allow  a commander  to  fire  first  if  he  feels  there  is  strong  enough 
indication  an  enemy  intends  to  attack. 

Without  this  provision  for  firing  first,  a fair  exercise  of  the  decision 
aids  could  not  have  been  achieved  since  a subject  would  not  have  decided  to 
attack  unless  fired  upon.  Obviously,  at  this  point,  the  DM  need  not  refer 
to  the  aids  studied  in  this  experiment. 

The  briefing  continued  with  a discussion  of  the  inputs  required  to 
the  system's  mathematical  model.  Likelihood  ratios  for  threat  categories 
within  generic  data  classes  constituted  the  first  of  these  inputs.  After 
describing  the  almost  overwhelming  problem  an  unaided  human  has  in  inter- 
preting the  mass  of  data  arriving  at  the  task  force  command  center,  the 
importance  of  anticipating  data  and  its  implication  concerning  Red  intent 
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was  stressed.  Rapid  processing  of  such  data  in  a manner  consistent  with 
the  laws  of  probability  to  achieve  reassessment  of  Red  intent  can  be  done 
with  computer  aids,  but  they  require,  as  inputs,  numerical  expressions  of 
subjective  understandings  of  data  implications.  The  concept  of  a likeli- 
hood ratio  was  then  introduced  with  specific  numerical  examples  based 
upon  the  briefer's  understanding  of  the  world.  Subjects  were  then  given 
a list  (Appendix  C)  of  the  generic  data  classes  (Red  threat  indicators) 
used  in  the  experiment  and  requested  to  supply  likelihoods  for  this  list 
that  best  captured  their  subjective  estimate  that  each  data  classification 
might  be  observed,  given  that  any  particular  hypothesis  about  the  enemy's 
intent  be  true.  As  a starting  point,  the  list  contained  likelihoods 
provided  by  the  briefer.  It  had  been  determined  in  planning  the  experiment 
that  an  individual  previously  unfamiliar  with  the  likelihood  concept  was 
more  comfortable  in  making  such  estimates  if  an  example  of  values  was 
provided  as  a starting  point.  A subject  was  allowed  as  much  time  as 
necessary  to  develop  his  likelihood  set,  with  30  minutes  being  approximately 
the  average  time  taken.  Likelihoods  were  estimated  by  first  selecting  per- 
centages within  each  intent  column  which  the  subject  thought  appropriate 
and  which  added  to  100  percent.  Then  ratios  of  these  percentages  across  the 
various  enemy  intents  (the  likelihood  ratios)  were  checked  and  adjusted 
to  capture  the  subject's  feelings.  The  process  was  repeated  until  a set 
of  numbers  had  finally  been  obtained  such  that  percentages  within  columns 
added  to  100  percent  and  likelihoods  across  columns  expressed  the  subject's 
understanding  of  the  relation  between  data  events  and  Red  intent. 

The  briefing  continued  with  a discussion  of  how  one  might  rank  order 
the  set  of  Blue  act  and  Red  intent  combinations  on  the  basis  of  one  combina- 
tion seeming,  all  things  considered,  to  be  better  than  another.  The  briefer's 
orderings,  for  both  the  one  CV  and  three  CV  cases,  were  supplied  as  examples, 
and  then  each  subject  was  asked  to  determine  his  own  orderings.  Subjects 
were  then  asked  to  supply  numerical  values,  on  a scale  of  0 to  -100,  to 
these  orderings,  with  0 being  at  the  "best"  end  of  the  scale  and  -100  at 
the  "worst."  It  was  stressed  that  value  sets  for  the  one  CV  and  three  CV 
cases  were  not  independent.  If,  for  instance,  a value  assigned  to  an  act 
and  intent  combination  in  the  three  CV  case  fell  between  two  values  assigned 
in  the  one  CV  case,  the  subject  was  to  verify  that  this  was  consistent  with 
his  subjective  ordering  of  the  three  act  and  intent  combinations  involved. 

The  result  of  this  numerical  scaling  was  two  utility  matrices,  one  for  each 
of  the  risk  cases. 

At  this  point  in  the  briefing,  a 7r-BRLT-like  theoretical  extrapolation 
technique  was  employed  to  extract  two  additional  matrices  incorporating 
the  DM's  risk  preference  in  the  one  CV  and  three  CV  cases. 

This  involved  determining  the  value  of  an  equal  (50-50)  gamble  between 
the  "best"  and  "worst"  outcome  combinations,  as  determined  in  the  above 
rank  ordering.  The  participant  was  asked  to  locate  a value  on  the  0 to  -100 
scale.  This  value  was  to  be  selected  so  that  the  participant  would  be  in- 
different between  receiving  an  outcome  with  that  value  for  certain  or 
receiving  an  outcome  determined  by  the  equal  probability  gamble  between 
the  best  and  the  worst  outcomes.  The  value  so  determined  was  used  to  solve 
for  the  exponent  0 in  the  power  function 
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Y = f (X)  = aX°, 

where  a is  a constant  selected  so  that  the  function  ranges  between  0 
and  -2000,  Y_  = -1000  at  the  point  where  = value  of  Indifference.  Under 
the  assumptions  of  substitutability  and  indifference,  as  well  as  the 
psychometric  power  function  itself,  a curve  was  produced  which  roughly 
described  the  participant's  utility  function  and  incorporated  some  aspect 
of  the  individual's  attitude  toward  a risky  choice.  The  function  was 
used  to  transform  values  in  the  original  matrix  to  form  a risk-incor- 
porating utility  matrix.  An  indifference  value  estimation  and  the  sub- 
sequent matrix  transformation  were  performed  for  both  the  one  CV  case 
and  the  three  CV  case. 

The  concept  of  an  act's  expected  utility  was  then  introduced  via  a 
specific  numerical  example  utilizing  utilities  drawn  from  the  briefer's 
one  CV  utility  table  and  a particular  probability  triplet  for  Red  intent. 
Special  care  was  taken  to  explain  that  expected  utility  is  one  dimension 
decision  makers  often  take  into  account  in  uncertain  situations,  but  it 
is  not  necessarily  the  only  one.  As  an  example,  it  was  pointed  out  that 
Las  Vegas  casinos  are  willing  to  base  their  operations  on  expected  value, 
whereas  the  thousands  of  players  who  visit  these  casinos  each  year  must 
be  considering  dimensions  in  addition  to  expected  value,  since  the  expected 
values  of  games  played  favor  the  casinos. 

After  careful  description  of  how  each  aid — triangle  and  bar  graph — 
presents  concepts  developed  in  the  brief,  subjects  were  given  a "hands-on" 
introduction  to  the  two  aids  at  NAVPERSRANDCEN ' s PDP-12  computer  facility. 
This  concluded  the  experiment's  briefing  portion. 

Subjects  returned  on  an  individual  basis,  usually  within  48  hours  of 
the  briefing,  for  the  experiment's  data-gathering  phase.  After  a brief 
review  of  his  required  tasks,  the  DM  was  assigned  two  scenarios  under  a 
particular  risk  condition  (one  CV  or  three  CV)  and  asked  to  proceed 
through  the  scenarios  utilizing  one  of  the  decision  aids  (triangle  or  bar 
graph)  or  no  aid.  When  the  two  scenarios  were  completed,  the  subject  was 
offered  the  opportunity  for  a brief  rest  break,  and  then  the  process  was 
repeated  until  the  subject  had  seen  triangle,  bar  graph,  and  no-aid  condi- 
tions. Decision  aids  were  provided  with  the  subject's  conditional  likeli- 
hoods and  appropriate  non-ir-BRLT  utility  matrix  elicited  previously.  When 
this  portion  of  the  data  gathering  was  concluded,  a subject  was  run  through 
two  additional  scenarios  utilizing  the  triangle  with  a tt-BRLT  matrix  of 
risk-incorporating  utilities  appropriate  for  the  risk  condition  assigned. 

At  the  beginning  of  each  scenario,  the  DM  was  provided  a summary  of 
own  and  enemy  force  activity  along  with,  when  the  triangle  or  bar  graph 
was  to  be  used,  an  appropriate  prior  probability  assessment  of  Red  intent. 
The  DM  then  turned  to  the  first  page  of  the  scenario  folder,  read  the 
report  describing  enemy  activity,  determined  which  threat  classification 
best  identified  the  data  event,  and  entered  the  classification  identifica- 
tion into  the  terminal.  If  a decision  aid  was  being  used,  the  DM's  own 
predetermined  likelihoods  associated  with  the  data  class  were  returned  on 
the  display,  and  the  DM  was  given  an  opportunity  to  adjust  these  likelihoods 
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before  the  aid  used  them  to  revise  prior  probabilities  about  enemy's 
intent.  If  he  decided  to  make  a likelihood  change,  it  could  either  be 
a permanent  or  a temporary  change.  With  a permanent  change,  the  new  set 
of  likelihoods  entered  replaced  the  predetermined  likelihoods  until  another 
permanent  change  was  made  to  the  set  or  until  the  end  of  the  scenario, 
whichever  occurred  first.  At  the  conclusion  of  a scenario,  the  pre- 
determined likelihoods  were  once  again  in  effect.  With  a temporary  likeli- 
hood change,  the  new  set  of  likelihoods  replaced  the  existing  set  for  just 
that  particular  page  of  the  scenario.  Upon  revision,  the  graphic  display 
was  updated  and  posterior  probabilities  shown  to  reflect  the  data  event's 
effect . 

The  DM  then  had  an  opportunity  to  examine  the  display  and  make  an 
estimate,  based  upon  all  available  information,  as  to  enemy's  intent. 

The  DM  was  then  asked  for  an  action  recommendation  for  his  own  force. 

He  then  had  to  rank  on  a scale  from  1 ("not  very  confident")  to  5 ("extremely 
confident")  how  confident  he  was  in  his  decisions  about  threat  assessment 
and  act  response.  Once  this  was  done,  a signal  was  given  for  the  DM  to 
proceed  to  the  second  data  event.  The  process  was  repeated  until  all  20 
data  events  of  the  scenario  had  been  presented. 

When  the  subject  had  completed  eight  scenarios,  a second  ir-BRLT 
technique  was  employed  to  once  again  estimate  utilities  that  reflected 
the  individual's  attitude  toward  risk.  This  was  accomplished  by  having 
the  individual  adjust  the  probability  components  of  a two-outcome  gamble 
so  that  he  would  be  indifferent  to  receiving  the  outcome  determined  by 
this  gamble  and  receiving  the  outcome  determined  by  a two-outcome  reference 
gamble.  The  reference  gamble  involved  a 99  percent  chance  of  receiving 
what  the  individual  had  judged  to  be  the  best  Blue  act  and  Red  intent  outcome 
combination  and  a 1 percent  chance  of  receiving  the  worst  outcome.  The 
gambles  that  the  participant  adjusted  involved  a it  percent  chance  of  receiving 
the  best  outcome  and  a (100-rt)  percent  chance  of  receiving  one  of  the  remaining 
outcomes.  The  participant  estimated  a TT-indif f erence  value  for  each  of 
the  remaining  act  and  intent  combinations  between  the  rank-ordered  "best" 
and  "worst."  By  this  procedure,  an  equivalence  was  established  which  implicitly 
determined  the  subject's  utility  for  each  of  the  outcomes  (Raiffa,  1968). 

This  second  estimation  of  risk-incorporating  utilities,  though  a procedure 
more  directly  derivable  from  the  utility  axioms  of  von  Neumann  and  Morganstern 
(1953)  than  the  first,  was  more  difficult  to  implement  within  the  framework 
of  the  subject  matter  and  experimental  task  of  our  study.  The  first  procedure 
required  the  subject  to  estimate  only  a single  indifference  value — an  important 
consideration  when  time  is  of  the  essence.  When  time  permitted,  the  second 
procedure  was  done  as  a corroborating  check  of  the  estimation  procedure. 


Finally,  the  subject's  participation  concluded  with  administration  of 
the  questionnaire  which  recorded  his  reactions  to  the  experiment. 


The  software  written  for  the  three  experimental  conditions  stored  all 
the  necessary  information  on  magnetic  tape  for  reconstructing  the  subject's 
display  at  any  point  in  the  scenario.  These  data  included  scenario  and  data 
identification,  the  subject's  classification  of  the  data,  any  modification 
of  the  likelihoods  he  introduced,  his  assessment  of  enemy  intent,  the  act 
that  he  selected,  and  the  confidence  ranking  he  gave  his  decision.  These 
data  were  transferred  to  a master  tape  and  used  to  produce  a "hard  copy" 
summary  at  the  end  of  each  session. 


RESULTS  AND  DISCUSSION 


Issues  of  Risk 

The  central  purpose  of  this  study  was  to  investigate  the  significance 
of  perceived  risk  in  decision  making  by  the  TFC.  The  hypothesis  was  tested 
by  making  certain  dimensions  of  the  risk  associated  with  each  alternative 
in  a decision  more  explicit.  If  the  decision  maker  utilizes  this  added 
information  in  a way  that  suggests  that  he  is  concerned  with  risk,  then  risk 
is  a factor  in  his  decision. 

Risk  as  an  Independent  Variable 

The  attempt  to  increase  perceived  risk  by  varying  the  number  of  CV's 
composing  the  task  force  met  with  limited  success.  This  was  evidenced  by 
the  fact  that  subjects  disagreed  with  the  4th  questionnaire  item,  which 
stated  that  it  was  more  important  to  make  the  correct  decision  when  there 
were  three  CV's  present  rather  than  one  CV.  This  statement  was  in  accord 
with  our  intention  that  the  one  CV  task  force  situation  be  perceived  as  one 
of  higher  risk  and  therefore  requiring  greater  need  for  the  correct  decision. 

As  indicated  previously  the  subjects  reported  their  attitude  toward  each 
item  by  choosing  a response  along  a five-point  Likert  scale  in  which  "1" 
indicated  strong  disagreement  with  the  statement  and  "5,"  strong  agreement. 

The  median  response  for  the  4th  item  was  "2,"  and  was  given  by  seven  out  of 
the  twelve  subjects.  However,  when  the  risk  estimation  procedures  were  used  to 
assess  utilities  in  a lottery  or  gaming  situation,  the  resulting  utility 
functions  for  the  high  and  low  risk  conditions  were  quite  inconsistent.  The 
value  of  the  exponent  0 in  the  power  function,  Y = aX  , which  describes  the 
utility  function  is  an  index  of  the  subject's  risk  preference.  Figure  7 
illustrates  the  curve.  If  0 is  described  by  the  expression  for  various 
values  of  0 greater  than  1,  then  the  utility  function  is  convex  in  quadrant 
III  and  indicates  a preference  toward  risk.  When  the  curves  are  convex,  an 
outcome  of  -50  (i.e.,  half  the  range  possible)  on  the  abscissa  is  associated 
with  an  amount  that  is  less  than  half  the  range  possible  on  the  ordinate.  If 
0 is  less  than  1,  the  curve  is  concave  and  indicates  an  aversion  to  risk. 

When  the  curves  are  concave,  an  outcome  of  -50  on  the  abscissa  is  associated 
with  an  amount  that  is  more  than  half  the  range  possible  on  the  ordinate. 

Table  2 presents  the  estimated  0's  for  the  subjects  under  both  risk  conditions. 
These  values  were  determined  by  fitting  the  power  expression  through  the 
single  point  obtained  from  the  subject  during  the  procedure  for  estimating 
risk  incorporating  utilities.  By  using  0 as  an  index  of  attitude  toward  risk, 
it  was  readily  apparent  that  large  individual  differences  existed  in  the 
manner  in  which  the  subjects  were  affected  by  the  risk  manipulation.  Seven 
of  the  twelve  subjects  were  more  risk  averse  under  the  high  risk  condition. 

The  remaining  five  subjects  were  more  risk  preferenced  under  the  high  risk 
condition. 
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Figure  / 


Table  2 


Estimated  0 for  the  Risk-Incorporating 
Utility  Function 


Subject 

High  Risk  (1CV) 

Low  Risk  (3CV) 

la 

4.60 

1.16 

2b 

.33 

2.13 

3b 

1.36 

2.27 

4b 

.29 

4.92 

5a 

13.51 

.30 

6a 

13.51 

.17 

7b 

.87 

1.98 

8b 

1.60 

2.41 

9b 

1.16 

6.58 

10a 

2.41 

1.00 

lla 

3.49 

2.41 

12b 

.72 

.97 

Classified  as  risk  preferenced 
^Classified  as  risk  averse 
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In  order  to  assess  the  behavioral  effects  of  the  risk  manipulation, 
it  was  necessary  to  establish  a performance  measure.  To  evaluate  the  per- 
formance of  a subject  and/or  aiding  condition,  a so-called  "batting  average" 
or  "outcome"  score  was  used.  The  manner  in  which  this  score  was  determined 
is  given  in  detail  in  Appendix  D.  Suffice  it  to  say  at  this  point  that  the 
performance  score  reflected  the  ability  of  the  subject  to  discern  the  enemy 
intent  in  a timely  fashion. 

The  subjects'  data  were  analyzed  by  a balanced  incomplete  block 
ANOVA.  The  analysis  was  first  conducted  upon  the  performance  scores  computed 
by  using  each  individual's  own  non-risk  incorporating  utility  matrix.  A 
subject's  score  in  each  condition  was  taken  to  be  the  sum  of  his  scores 
on  all  the  scenarios  seen  under  that  condition.  Considering  all  the  subjects 
combined,  no  significant  difference  was  found  among  the  performances  using 
any  of  the  aids  and  under  any  risk  condition  (F  = .147;  df  = 5,  19).  The 
performance  scores  are  given  in  terms  of  a negative  score  because  the 
utilities  were  measured  as  relative  costs  for  each  outcome.  The  mean  per- 
formance scores  for  each  condition  are  given  in  Table  3.  Better  performance 
is  given  as  a less  negative  score.  The  means  for  each  condition  are  also 
given  in  Table  3.  /^.though  the  differences  are  not  reliable,  it  was  sur- 
prising to  see  that  performance  using  the  aids  was  lower  than  the  performance 
in  the  no-aid  control.  However,  if  the  subjects  were  grouped  on  the  basis 
of  whether  their  0 indicated  that  they  were  more  risk  averse  or  risk  pre- 
ferenced  under  the  high  risk  condition,  a look  at  the  subgroup  means  reveals 
an  interesting  result.  In  order  to  te^t  the  effect  of  the  aiding  conditions 
upon  performance  for  the  risk  averse  subgroup,  a Friedman  two-way  analysis 
of  variance  of  ranks  was  performed.  The  results  showed  the  overall  dif- 
ference between  aiding  conditions  to  be  slight  (X2r,  2df  = 4. 57;  p .112). 

A post  hoc  comparison  of  the  treatments  against  the  control  showed  the  per- 
formance using  the  DDI  display  to  be  significantly  (r*  =8;  p <_  .05)  better 
than  the  control.  Performance  using  the  NAVPERSRANDCEN  display  was  somewhere 
between  and  not  significantly  different  from  the  control.  The  risk  pre- 
ferenced  subjects  performed  best  under  the  control  condition,  surpassing  the 
performance  of  the  risk  averse  subjects  at  their  best.  However,  the  overall 
difference  was  not  significant  (x2r  2df  = 1-60;  p <_  .522)  and  a comparison 
of  the  treatment  means  with  the  control  showed  no  significant  differences 
either.  These  results  indicate  that  only  risk  averse  individuals  benefited 
from  information  supplied  by  the  DDI  aid.  The  findings  suggest  that  (1)  the 
hypothesis  that  performance  can  be  improved  by  making  certain  aspects  of 
risk  explicit  is  not  true  and  (2)  risk  preferenced  individuals  may  actually 
have  been  slightly  hindered  by  the  presence  of  an  aid.  The  question  arises 
as  to  why  the  same  information  made  available  to  two  groups  would  benefit 
the  group  which  wishes  to  avoid  risk  and  yet  would  hinder  the  group  which 
prefers  risk.  One  speculative,  but  possible,  explanation  arises  from 
the  demand  characteristics  of  the  study.  The  briefing  placed  considerable 
emphasis  upon  the  role  of  SEU  in  selecting  from  amongst  alternatives.  The 
aids,  as  designed,  prominently  display  SEU  information.  If  SEU  information  is 
not  normally  a major  factor  in  the  decision  process,  an  attempt  to  incorporate 
it  could  be  disruptive.  To  test  this  possibility,  another  performance  score 
was  calculated  for  the  two  subgroups.  Instead  of  scoring  the  act  that  the 
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subject  actually  selected,  we  scored  the  act  which  the  aid  recommended  on 
each  trial.  In  effect,  this  was  a scoring  of  the  system's  performance.  This 
scoring  was  done  only  for  the  DDI  triangle  display  because  it  was  the  only 
display  in  which  a specific  act  recommendation  was  made.  The  result  from 
this  scoring  of  the  aid's  recommendation  is  shown  in  Table  3 alongside 
scores  based  on  the  actual  selected  act.  Note  that  the  mean  "aid  alone" 
performance  for  the  risk  preferenced  group  is  considerably  worse  than  the 
mean  performance  for  the  individual  using  the  aid  ("aid  and  individual"). 

This  difference  is  consider?11  larger  than  the  corresponding  difference 
for  the  risk  averse  group.  .a  indication  here  is  that  individuals  in  the 
risk  preferenced  group  were  introducing  some  factor  or  factors  other  than 
maximum  SEU  into  the  decision  process  and  were  doing  it  to  a greater  extent 
than  the  individuals  in  the  risk  averse  group.  Whatever  these  factors  are, 
they  may  ordinarily  have  been  these  individuals'  primary  factors  for 
selecting  acts.  In  the  control  condition,  they  were  not  "burdened"  by  the 
display  along  with  its  recommendation  and,  consequently,  performed  better. 

Since  the  differences  shown  in  Table  3 could  be  capitalizing  upon 
the  differences  in  the  utility  matrices  given  by  the  individuals,  the 
analysis  was  repeated  using  an  average  utility  matrix.  A matrix  consisting 
of  the  mean  non-risk  incorporating  utilities  for  each  act/intent  outcome 
was  determined.  This  average  matrix  was  then  used  to  score  both  the  per- 
formance of  the  individuals  (using  the  aids)  and  the  performance  of  the  aid 
alone.  The  means  of  this  are  given  in  Table  4.  Analyses  of  these  scores 
reveal  essentially  the  same  scoring  results  and  implications  as  the  analyses 
of  the  scores  in  Table  3. 
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Mean  Performance  Scores  by  Conditions 


Mean  Performance  Scores  by  Condition 


The  Dimensions  of  Choice 


iiy  making  certain  aspects  of  the  action  alternatives  more  explicit 
in  the  NAVPERSRANDCEN  bar  graph  display,  we  had  anticipated  a decline  in 
the  frequency  of  times  that  the  act  the  subject  selected  corresponded  with 
the  act  which  had  the  maximum  SEU.  The  percentage  of  times  that  a subject 
agreed  with  the  maximum  SEU  act  is  given  in  Table  5.  As  might  be  expected, 
the  frequencies  were  higher  for  the  conditions  in  which  tne  aids  were  present. 
Contrary  to  our  expectations,  the  frequency  of  subject  agreement  with  the 
maximum  SEU  was  higher  for  the  bar  graph  condition  than  for  the  DDI  triangle 
condition.  An  interaction  between  decision  aids  and  risk  conditions  is  suggested 
by  the  data.  When  using  the  DDI  triangle,  subjects  agreed  with  the  aid's  recom- 
mendations slightly  more  often  under  the  high  risk  condition  than  under  the 
low  risk  condition.  The  situation  was  reversed  when  using  the  NAVPERSRANDCEN 
bar  graph  display.  Subjects  showed  a strong  tendency  to  maximize  SEU  in  the 
low  risk  condition  when  such  behavior  was  most  appropriate.  When  a high  risk 
condition  occurred,  the  frequency  of  SEU  maximization  decreased  as  predicted. 

This  effect  occurred  only  in  the  bar  graph  condition  and  supports  the  notion 
that  the  subjects  were  incorporating  risk  into  their  judgments.  Once  again, 
the  interaction  is  in  line  with  the  hypothesis  that  the  subject  is  utilizing 
explicit  information  in  addition  to  SEU  and  is  more  likely  to  do  so  under 
conditions  of  high  risk.  However,  the  difference  in  the  amount  of  subject 
agreement  with  maximum  SEU  between  the  low  risk  case  under  the  triangle 
condition  and  low  risk  under  the  bar  graph  condition  (66.2%  vs.  81.2%)  is 
not  explainable  by  the  hypothesis.  If  information  about  risk  is  not  par- 
ticularly relevant,  as  one  would  expect  to  be  the  case  in  the  low  risk 
condition,  then  dependence  upon  the  maximum  SEU  recommendation  should  be 
approximately  the  same  under  the  DDI  triangle  as  under  the  bar  graph  dis- 
play. If  dependence  under  one  of  the  conditions  should  be  greater,  it  ought 
to  be  the  dependence  under  the  DDI  triangle  because  of  the  ease  with  which 
the  maximum  SEU  act  is  interpreted.  Grouping  the  percentages  according  to 
risk  subgroups  shows  that  the  risk  preferenced  subgroup  selected  the  maximum 
SEU  act  more  often  than  the  risk  averse  subgroup,  even  in  the  control  condition. 
This  may  seem  completely  counter  to  most  of  the  previously  discussed  results, 
which  show  an  improvement  in  performance  for  the  risk  averse  subgroup  using 
the  aids.  Now  we  are  saying  that  the  very  thing  which  the  aid  presented  to 
the  subject  was  used  more  often  by  the  subjects  who  did  not  benefit  from  the 
aid.  Closer  investigation  of  the  trial  by  trial  data  revealed  that  there 
were  many  occasions  in  which  one  alternative  was  so  superior  in  terms  of  SEU 
that  additional  information  was  completely  unnecessary.  Occurrences  of  such 
instances  could  have  reduced  the  observed  effect  of  making  information  about 
risk  more  explicit. 

What  were  the  factors  in  addition  to  SEU  that  the  subject  might  have  been 
using?  One  obvious  candidate  was  the  variance  associated  with  each  alternative. 
When  an  alternative  has  the  potential  of  resulting  in  a wide  range  of  outcome 
(high  variance),  the  selection  of  that  alternative  does  not  give  the  decision 
maker  as  much  assurance  about  the  consequences  of  his  action  as  does  the 
selection  of  an  alternative  with  lower  variance.  Thus,  it  would  be  expected 
that  the  subject  would  be  more  likely  to  select  the  alternative  with  the 
minimum  variance  in  situations  of  high  risk. 


Percentage  of  Trials  Subject  Agreed  With  Maximum  SEU  Act 
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To  examine  tills,  an  analysis  was  performed  to  determine  the  percentage 
of  trials  that  the  act  with  the  minimum  variance  was  the  act  that  the  subject 
selected.  The  results  are  given  in  Table  6.  Once  again,  the  overall  means 
showed  no  difference  in  the  aiding  conditions.  The  mean  percentage  for  the 
DDl  low  risk  condition  was  much  lower  than  the  percentage  for  any  of  the 
other  conditions.  A closer  inspection  of  the  data  showed  that  the  percentages 
may  be  inflated  because  of  the  occurrence  of  so  many  instances  in  which  the 
maximum  SEU  act  and  the  minimum  variance  act  were  the  same  act.  This 
coincidence,  however,  did  not  occur  as  often  in  the  DDI  low  risk  condition. 

A second  analysis  was  performed  to  avoid  the  confounding.  The  figures  in 
parentheses  are  the  percent  of  trials  in  which  the  maximum  SEU  act  was 
different  from  the  minimum  variance  act  and  the  subject  selected  the  act 
with  minimum  variance.  The  results  indicate  no  overall  difference  among 
the  aiding  conditions.  However,  the  parenthesized  figures  show  an  apparent 
interaction  between  the  aiding  conditions  and  the  risk  manipulation. 

Although  the  subjects  predominately  select  the  acts  which  have  the  maximum 
SEU,  this  predominance  was  less  under  high  risk  when  the  subjects  were 
using  the  DDl  triangle,  about  the  same  under  high  and  low  risk  when  using 
the  NAVPERSRANDCEN  bar  graph,  and  greater  under  high  risk  when  using  no  aid 
at  all.  This  result  was  not  expected  since  variance  information  was  not 
explicitly  available  in  the  DDI  triangle. 


Percentage  of  Trials  Subject  Agreed  With  Minimum  Variance 


The  predominance  of  maximum  SEU  was  further  supported  by  an  attempt 
to  use  a weighted  linear  combination  of  SEU  and  variance  to  predict  the 
subjects'  selection  of  acts.  On  each  trial,  the  SEU  and  variance  of 
each  action  alternative  was  standardized  with  respect  to  the  SEU's  and 
variances  of  the  other  acts  on  that  trial.  These  two  standard  scores 
were  weighted  and  summed  to  form  a "desirability"  or  "preference"  score 
for  each  alternative: 

Desirability  of  act  i = a x STD  SEU^  -Ex  STD  VARIANCE^ 
where,  -1.0  (u,  B)  1.0. 

The  alternative  witli  the  maximum  desirability  score  was  taken  to  be  the 
model's  predicted  act  on  that  trial.  The  weights  u and  B for  the  linear 
combination  were  estimated  by  using  a parameter  searching  program  called 
STEI’IT  (Chandler,  1969)  to  maximize  the  number  of  correct  predictions  by 
the  linear  model.  The  fit  was  performed  only  for  the  high  risk  conditions. 
The  resulting  best  fit  weighting  coefficients  for  SEU  and  variance  are 
shown  below: 

DPI  Triangle 

Di  = (.74)  STD  SEL^  - (.044)  STD  VARIANCE^ 

NAVPERSRANDCEN  Bar  Graph 


Di  = (1.0)  STD  SEUi  - (-1.0)  STD  VARIANCE A 
Control 

Dt  = (1.0)  STD  SEUi  “ (~-13)  STD  VARIANCEi 

The  relative  sizes  of  the  weights  indicate  the  relative  contribution  that 
the  respective  factors,  SEU  and  variance,  had  in  the  establishment  of  an 
act's  desirability  and  thence  act  selection.  The  weights  emphasize  the 
small  role  played  by  variance  under  the  DDI  triangle  condition  and  under 
the  control  condition.  The  role  of  variance  appeared  to  be  much  greater 
under  the  NAVPERSRANDCEN  bar  graph  condition.  In  this  condition,  the 
combination  of  SEU  and  variance  led  to  a substantially  greater  number  of 
correctly  predicted  acts  than  did  the  maximum  SEU  model  (179  corrects 
versus  163  corrects  out  of  240  trials). 


Finally,  the  possibility  that  some  type  of  minimax  strategy  might  have 
been  employed  was  examined.  The  most  negative  outcome  possible  for  each 
alternative  on  each  trial  was  determined.  The  utility  associated  with  that 
outcome  was  weighted  by  the  probability  that  that  outcome  would  in  fact 
occur.  The  act  which  had  the  least  negative  product  was  deemed  the  minimax 
act.  When  the  maximum  SEU  act,  minimum  variance  act,  and  the  minimax  act 
differed,  the  selection  of  the  subject  could  reveal  his  performance  for  a 
strategy.  The  proportions  of  the  time  that  the  subjects  were  faced  witli 
this  choice  and  picked  tiie  minimax  act  were  computed.  Unfortunately,  the 
proportions  found  were  based  upon,  at  most  15  possible  occurrences. 

Because  of  the  design  of  the  experiment  and  these  few  observations,  little 
was  concluded  about  the  implications  of  the  proportions. 


30 


The  Usefulness  of  "Canned"  Likelihood  Ratios 

The  DDI  decision  aid  uses  a so-called  "canned"  or  predetermined 
likelihood  approach  in  which  the  decision  makers  estimate  the  diagnostic 
impact  of  potential  data  events  well  in  advance  of  the  actual  observation 
of  these  events.  Typically,  these  estimations  are  made  in  the  planning 
stages  of  the  operation,  a period  during  which  lists  of  possible  data 
events  and  their  implications  are  generated.  The  advantage  of  the 
canned  likelihood  ratio  approach  is  obviously  the  quick  response  time 
its  use  makes  possible.  A system  using  canned  likelihood  ratios  can 
operate  in  a semiautomatic  mode  with  respect  to  the  data.  When  the 
occurrence  of  a particular  data  event  is  observed,  its  occurrence  is 
indicated  to  the  system.  The  canned  likelihood  ratios  associated  with 
that  event  are  immediately  and  automatically  employed  to  update  the 
inference  algorithm,  Bayes'  Theorem.  Thus,  the  updating  of  the  posterior 
probabilities  occurs  rapidly  with  relatively  little  human  intervention. 

However,  the  predetermination  of  likelihood  ratios  does  present  certain 
problems.  In  this  section  of  the  paper,  we  will  discuss  two  of  these 
problems  in  some  detail  and  present  results  from  the  study  which  have 
bearing  upon  these  problems.  The  first  of  these  problems  to  be  discussed 
is  the  intersubject  reliability  or  consistency  of  the  subjects'  estimates 
of  the  likelihood  ratios.  In  this  study,  any  inconsistencies  in  likelihood 
ratio  estimates  led  to  different  posterior  probabilities  and,  possibly, 
different  decisions.  The  subject  always  begat  each  scenario  with  the  set 
of  likelihood  ratios  which  he  himself  estimated.  Formal  Bayesian  updating 
using  these  likelihoods  rather  than  some  prescribed  set  is  likely  to  be 
more  descriptive  of  the  DM's  own  cognitive  revision  and  make  him  feel 
"comfortable"  about  using  the  aid.  However,  if  the  goal  of  decision  aiding 
is  not  to  emulate  the  DM's  decision  process  but  to  optimize  his  decisions, 
it  becomes  imperative  to  conduct  the  Bayesian  revision  utilizing  a likeli- 
hood set  which  is  as  close  as  possible  to  the  true  one.  Of  course,  if  the 
DM's  own  likelihood  set  is  very  discrepant  from  this  true — or  nearly  true — 
set,  he  may  feel  uncomfortable  using  an  aid  which  makes  counter-intuitive 
suggestions . 

The  adequacy  of  using  a particular  DM's  likelihood  set  could  be  assessed 
if  it  could  be  compared  to  the  true  likelihood  set.  Unfortunately,  the  true 
set  is  rarely  obtainable.  Thus,  expert  subjective  estimates,  as  were  used 
in  this  study,  must  be  relied  upon.  When  the  intersubject  consistency  of 
such  estimates  is  high,  each  DM's  set  is  likely  to  be  a good  approximation 
of  the  true  set.  In  this  case,  it  is  probably  suitable  to  let  each  DM  use 
his  own  likelihood  set  during  Bayesian  revisions.  However,  when  this  inter- 
subject consistency  is  low,  it  may  be  necessary  to  obtain  a prescribed 
likelihood  set  by  a group  estimation  procedure. 

The  second  problem  is  that  of  conditional  dependence.  The  adequacy  of 
the  canned  likelihood  approach  is  in  question.  If  the  data  are  conditionally 
dependent,  the  canned  likelihood  ratios  estimated  in  advance  may  not  be 
appropriate  for  the  data  at  the  time  they  are  actually  received. 
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Before  statements  about  the  reliability  of  the  precanned  likeli- 
hood estimates  can  be  made,  it  is  necessary  to  develop  a measure  which  will 
allow  the  reliability  or  the  consistency  of  the  various  likelihood  estimators 
to  be  measured.  The  development  of  such  a measure  follows.  The  likelihood 
estimates  required  for  both  the  DDI  and  NAVPERSRANDCEN  aid  consist  of  a 
vector  of  three  conditional  probabilities  or  likelihoods.  Each  likelihood 
is  divided  by  the  sum  of  the  three  likelihoods.  The  effect  of  this  pre- 
liminary calculation  is  simply  to  normalize  the  likelihood  vector  so  that 
the  sum  of  its  three  elements  is  equal  to  one.  This  normalized  vector  can 
then  be  thought  of  as  defining  a point  in  a three-dimensional  probability 
space.  Technically,  these  normalized  likelihoods  are  posterior  probabilities 
computed  using  a uniform  prior  distribution.  If  the  likelihoods  for  each 
subject  are  plotted  in  three-space  in  this  normalized  fashion,  the  density 
of  the  swarm  of  points  so  formed  characterizes  the  consistency  of  the  various 
likelihood  estimates.  Because  these  likelihoods  are  normalized,  it  is  also 
the  case  that  these  points  all  fall  on  a plane  defined  by  the  sides  of  the 
DDI  triangle.  A plot  of  these  points  on  the  plane  forms  a scatter  diagram 
and  is  then  a pictorial  representation  of  the  variability  in  the  likelihood 
estimates,  described  in  terms  of  the  triangular  representation  used  by  DDI. 

A descriptive  measure  characterizing  each  subject's  variability  can  be 
developed  by  finding  the  centroid  of  this  swarm  of  points  in  the  DDI  tri- 
angle and  expressing  each  subject's  departure  from  that  centroid  in  terms 
of  absolute  distance.  The  centroid  defines  the  so-called  "average  subject" 
or  group  consensus,  and  the  departures  from  the  centroid  define  the  variability 
around  this  mean.  It  was  assumed  that  the  centroids  best  represented  the 
group  consensus  and  that  the  group  consensus  was  the  best  estimate  of  the 
true  likelihood  ratios.  Subject  agreement  with  the  group  centroids  was 
measured  as  the  mean  Euclidean  distance  between  the  subject's  likelihood 
ratios  and  the  group  centroids.  There  was  a considerable  difference  in 
this  measure  between  the  subject  whose  average  distance  was  closest  to  the 
group  centroids  and  the  subject  who  was  farthest  from  the  group  centroids. 

The  mean  distance  for  all  subjects  from  the  group  centroids  was  a probability 
distance  of  .139.  This  distance  is  interpretable  in  terms  of  the  surface 
contained  within  the  DDI  triangle.  It  means  that,  in  a triangular  space 
with  sides  equal  to  / 2 , the  average  subject  may  be  conceptualized  as  being 
.14  units  from  the  group  mean.  The  subject  whose  estimates  were  closest  in 
average  distance  to  the  group  centroids  had  a mean  absolute  deviation  from 
the  centroids  of  ,0B9,  while  the  subject  who  showed  the  most  disparity  from 
the  group  centroids  h.  cl  a mean  absolute  deviation  of  . 272.  Thus,  there  is 
approximately  a three-to-one  difference  between  the  subject  whose  estimates 
were  most  closely  captured  by  the  group  centroids  and  the  subject  whose 
estimates  were  least  well  captured.  This  large  difference  is  somewhat  dis- 
turbing since  it  was  anticipated  that  the  estimates  from  a group  of  experts 
would  be  very  closely  clustered  about  the  group  centroids.  The  size  of  the 
difference  suggests  considerable  subject  variability,  a variability  which  is 
undersirable  and  perhaps  reducible  by  some  type  of  group  likelihood  estimation 
procedure . 
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Another  interesting  aspect  of  consistency  results  from  examination 
of  likelihood  estimation  for  the  various  groups  of  data  classes.  In  this 
analysis,  rather  than  seeking  to  determine  how  far  a given  subject  is  from 
the  group  average,  we  investigate  how  much  variability  exists  between 
various  estimates  for  each  data  class  used  in  the  experiment.  The  results 
of  this  analysis  showed  considerable  differences  between  data  classes  in 
terms  of  how  consistently  the  subjects  assessed  likelihood  ratios  for  each 
class  of  data.  Examples  of  data  classes  where  there  was  relatively  low 
consistency  among  the  subjects'  likelihood  estimates  are  those  for  SS-N-3 
surface  platform  activity,  the  SS-N-2/SNSS-N-11  platform  activity,  SAM 
platform  activity,  and  the  AS-1/2/3  platform  activity.  The  average  devia- 
tions of  the  estimates  from  the  centroids  for  these  data  classes  were  .179, 
.174,  .191,  and  .191,  respectively.  In  contrast  to  the  relatively  high 
disagreement  for  these  data  classes,  there  were  other  data  classes  in  which 
the  subjects  showed  fairly  close  agreement.  Data  classes  dealing  with 
aircraft  activities  had  a mean  distance  from  the  centroid  of  .069  and  .087. 

For  indicators  having  to  do  with  missile  launching,  the  mean  distance  from 
the  centroid  was  .056  and  .072,  respectively.  Average  distance  from  cen- 
troids of  estimates  for  enemy  surveillance  activity  was  .069.  Clearly,  some 
data  classes  create  more  consistent  estimates  than  do  others,  and  the  degree 
of  this  consistency  seems  to  be  associated  with  the  type  of  data  class. 

Data  classes  involving  surface  or  subsurface  weapons  platforms  seem  to  be 
judged  less  consistently  than  data  classes  involving  either  aircraft,  missiles, 
or  surveillance. 

In  summary,  the  consistency  of  the  subjects'  canned  likelihood 
estimates  is  less  than  ideal,  and  it  might  be  advisable  to  investigate  the 
feasibility  of  increasing  the  reliability  by  some  group  estimation  procedures. 


Canned  Likelihoods  and  Conditional  Dependencies  Among  Data 


Another  major  obstacle  to  the  canned  likelihood  approach  is  the 
existence  of  conditional  dependencies  among  data.  Conditional  dependencies 
are  said  to  occur  when  the  interpretation  that  one  puts  upon  any  given  datum 
is  influenced  by  the  occurrence  or  nonoccurrence  of  previous  data.  For 
example,  if  a particular  datum  becomes  especially  significant  when  coupled 
with  other  data  previously  received,  then  the  data  in  question  are  said  to 
be  conditionally  dependent.  Decision-aiding  systems  which  employ  Bayes' 

Theorem  should  exploit  any  conditional  dependencies  that  are  present  in  the 
data.  Various  approaches  to  conditional  dependence  problems  have  been  employed. 
These  approaches  for  conditional  dependence  are  discussed  extensively  by 
Steiger  and  Gettys  (1973).  The  Semi-PIP  approach  employed  by  Gustafson  (1969) 
is  not  a viable  alternative  for  the  DDI  approach  as  it  precludes  canning  of 
likelihoods.  This  approach  is  impractical  if  the  patterns  of  conditional 
dependencies  are  extensive.  The  number  of  likelihood  estimates  required 
for  each  data,  conditioned  upon  the  previous  data  that  have  been  seen, 
rapidly  becomes  astronomical  in  number.  For  this  reason,  to  attempt  to 
anticipate  all  of  the  conditional  dependencies  is  not  a viable  alternative. 


Another  procedure,  termed  conditional-PIP,  can  be  readily  incorporated 
into  the  framework  of  the  DDI  concept,  although  there  is  some  question  of 
human  ability  to  estimate  conditional  likelihood  ratios  (Steiger  & Getty-, 
1973).  Using  a conditional-PIP  approach,  conditional  dependencies  art 
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incorporated  into  tlie  likelihood  ratios  by  adjusting  the  likelihoods  to 
reflect  Lhe  impact  of  previous  data.  This  adjustment  necessarily  involves 
human  intervention  to  generate  revised  likelihoods  that  incorporate  the 
patterns  of  conditional  dependencies  in  the  data.  If  the  subject  receives 
a datum  which  he  believes  is  conditionally  dependent  with  respect  to  data 
previously  received,  all  he  need  do  is  adjust  the  likelihoods  to  take  this 
conditional  dependency  into  account.  The  NAVPERSRANDCEN  software  allowed 
this  opportunity.  If  the  subject  believed  that  the  canned  likelihoods 
were  inappropriate,  he  could  modify  the  likelihoods  at  will.  The  new 
likelihoods,  which  incorporated  the  impact  or  moderating  effect  of  previous 
data,  were  then  utilized  by  the  Bayesian  system  in  lieu  of  the  canned 
estimates.  The  percentage  of  temporary  changes  for  all  subjects  and  all 
scenarios  was  18.53  percent.  Permanent  changes  occurred  with  a frequency 
of  2.28  percent.  The  greater  number  of  temporary  changes  suggests  a rather 
frequent  perception  of  conditional  dependencies. 

For  these  reasons,  it  is  of  considerable  interest  to  assess  in  more 
detail  the  extent  to  which  subjects  were  exploiting  conditional  dependencies 
in  their  data.  Another  suggestive  preliminary  result  was  obtained  by 
plotting  for  each  page  of  a scenario,  as  a function  of  the  total  number  of 
data  received  up  to  and  including  that  page,  the  cumulative  percentage  of 
times  decision  makers  intervened  and  made  temporary  changes  to  canned 
likelihood  values.  The  rationale  for  this  analysis  is  the  argument  that 
data  which  occur  late  in  a scenario  are  more  likely  to  be  conditionally 
dependent  upon  previous  data  than  those  that  occur  early  in  the  scenario. 

If  subjects  were  exploiting  conditional  dependencies,  then  they  should 
have  made  temporary  changes  in  the  likelihoods  to  reflect  these  conditional 
dependencies.  Therefore,  if  the  rate  at  which  they  made  these  changes 
increased  as  a scenario  progressed,  this  would  be  suggestive,  but  not 
conclusive,  evidence  for  conditional  dependencies  in  the  data.  Consequently, 
we  calculated  the  cumulative  percentage  of  temporary  changes  in  likelihoods 
for  each  page  of  each  scenario.  An  increase  in  the  percentage  of  times 
the  subjects  made  temporary  changes  as  a scenario  progressed  occurred  in 
four  scenarios.  The  remaining  scenarios  showed  no  such  clearcut  trends. 

These  preliminary  results  convinced  us  that  still  further  analysis  was 
just  if  led . 

A third  analysis  involved  identifying  data  that,  in  the  opinion 
of  the  scenario  writer,  were  conditionally  dependent  with  respect  to  other 
data.  If  the  likelihood  ratios  for  these  data  were  changed  in  a scenario, 
this  I act  was  noted.  On  the  24  occasions  when  these  data  were  not  preceded 
by  tin  conditioning  data,  the  subjects  made  a change  in  the  likelihoods 
8.11  percent  of  1 1 jo  time.  On  the  70  occasions  when  the  data  were  preceded 
by  conditioning  data,  changes  were  made  32  percent  of  the  time.  This  dif- 
fer one  t*  in  the  frequency  of  likelihood  changes  is  statistically  reliable 
(•  >.11,  p .05).  Botli  of  these  analyses  are  suggestive  of  conditional 

dep  one  | (*.  in  the  data  which  were  exploited  by  the  subjects. 

\1 though  tin  above  analyses  are  suggestive  of  conditional  dependencies 
• it. i,  there  were  actually  two  ways  that  a subject  in  our  experiment 
» dltlonnl  dependent:  ies  into  account.  The  first  technique  was 


34 


1 

E 4 


by  likelihood  changes.  The  second  technique  was  by  changing  the  threat 
classification.  If,  as  a result  of  previous  data,  the  decision  maker 
perceives  the  threat  of  the  current  datum  differently  and  changes  his 
classification,  then  this  also  is  evidence  for  conditional  dependence. 

Either  a change  in  the  likelihoods  or  change  in  threat  classification, 
therefore,  can  result  from  conditional  dependencies  that  the  subject 
detects  in  the  data.  An  analysis  which  taps  the  joint  effect  of  these 
two  techniques  for  incorporating  conditional  dependencies  in  the  data 
can  be  performed  by  examining  the  distribution  of  the  subjects'  likeli- 
hoods for  the  same  datum  in  various  scenarios.  Several  commonly  used 
data  items  which  occur  in  a number  of  scenarios  were  identified  and  the 
subjects'  likelihood  responses  to  these  data  were  obtained.  These  likeli- 
hoods were  subjected  to  a multivariate  analysis  of  variance.  This  analysis 
showed  that,  in  fact,  the  centroids  of  the  same  data  differed  in  a scenario- 
dependent  manner.  This  result  is  reliable  (F  = 1.6132,  p < .0658)  and 
strongly  suggests  that,  in  fact,  subjects  were  exploiting  conditional 
dependencies  present  in  the  data  either  by  making  a likelihood  change  or 
by  picking  different  threat  classifications. 

In  summary,  it  does  appear  feasible  to  can  likelihood  estimates 
in  advance  if  the  subjects  are  given  a means  for  modifying  these  likelihoods 
to  exploit  conditional  dependencies  and  for  incorporating  unanticipated  data 
into  the  scenarios.  A near-instantaneous  response  was  achieved  for  approxi- 
mately 80  percent  of  the  data,  the  remaining  20  percent  required  a relatively 
rapid  likelihood  estimate  for  further  processing. 

Problems  With  the  Inference  Model 

It  became  apparent  when  running  the  system  with  real  scenarios  that 
there  was  a difficulty  with  the  inference  model.  The  system  was  sometimes 
slow  to  respond  to  data  that  were  obviously  diagnostic.  An  example  of  this 
phenomenon  is  presented  below.  It  was  termed  the  "trapped  bug"  phenomenon 
by  the  researchers. 

The  Case  of  the  Trapped  Bug 

A very  clear  example  of  the  trapped  bug  problem  can  be  seen  in  the 
difficulties  encountered  by  subject  6 while  proceeding  through  one  of  the 
scenarios.  After  classification  of  datum  #15  in  this  scenario,  the  system 
estimate  of  Red  intent  had  been  driven  to 

P (Routine)  = .001  . . . 

P (Harass)  = .998  . . . 

P (Attack)  = .000  . . . , and  the  subject's  analysis  of  Red  intent 

was  to  classify  it  as  harassment.  In  fact,  all  subjects  judged  Red's 

likely  intent  to  be  harassment  after  seeing  the  15th  datum  of  this  particular 
scenario. 

An  unclassified  version  of  the  remaining  five  (16-20)  data  items 
in  the  scenario  is  shown  in  Table  7.  These  items  occurred  over  a 3 minute 
period.  A summary  of  the  subject's  analysis  and  system  performance  during 
this  period  appears  in  Table  8. 
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Table  7 

Data  Items  Associated  With  a Trapped  Bug 


Item 

Description 

16 

Jamming  of  air  search 

radars  begins. 

17 

Chaff 

usage  and 

additional  jamming  occurs. 

18 

Target 

platform 

radar 

signal  detected. 

19 

Another  targeting  platform  radar 

signal  detected. 

20 

Air-to 

-surface  missile 

targeting 

radar  detected. 

Table  i 

B 

Subject  Analysis  and 

System 

Performance  Associated  With  a ' 

Trapped  Bug 

Datum 

Int 

Dec  L(R) 

L(H)  L (A) 

P(R)  P (H) 

P (A) 

16 

A 

A 1 

5 

50 

.00210  .999698 

.000092 

17 

A 

A 1 

5 

50 

.000043  .999865 

.000920 

18 

A 

A 1 

5 

50 

.000008  .990892 

.009100 

19 

A 

A 1 

1 

96 

.000005  .531043 

.468952 

20 

A 

A ] 

5 

50 

.0000002  .101721 

.398279 

Legend:  Int 

= 

subject's  analysis  of  Red  intent 

Dec 

= 

subject's  decision  on  own  force  action 

R 

= 

routine 

H 

= 

harass 

A 

= 

attack 

L( . ) 

= 

likelihood  assigned  Red  intent 

P(.) 

= 

system  probability  estimate  of  Red  intent 

It  can  be  seen  that,  even  though  the  subject  obviously  felt  he  was 
very  likely  in  an  attack  situation  from  datum  16  on  and  input  likelihoods 
were  consistent  with  this  feeling,  system  probability  estimates  were  very 
slow  to  move  towards  an  attack  indication.  In  other  words,  system  indica- 
tion of  the  existing  tactical  situation  was  not  congruent  with  the  strongly 
based  perception  of  that  situation  by  the  subject. 

Overall,  in  roughly  40  percent  of  the  cases  that  an  attack  scenario 
was  run  in  the  experiment,  the  bug  trapping  problem  appeared. 

This  lack  of  responsiveness  apparently  occurs  because  the  inference 
model  used  by  DDI  is  not  appropriate  for  the  TFC  situation.  This  model 
assumes  the  inference  process  is  "stationary,"  that  is,  the  data  generating 
process  remains  constant  throughout  the  scenario.  In  fact,  the  scenarios 
which  we  used  are  nonstationary  and  the  data  generating  process  sometimes 
changed  throughout  the  course  of  the  scenario.  In  other  words,  the  enemy 
may  not  exhibit  any  overt  hostile  acts  until  well  into  the  scenario. 

Because  these  hostile  acts  do  not  occur  until  relatively  late  in  the 
scenario,  the  system  concludes  that  the  probability  of  attack  is  vanishingly 
small  because  all  data  prior  to  the  receipt  of  these  hostile  acts  suggested 
harassment.  At  some  time  during  the  scenario,  the  Red  forces  change  the 
data  generating  process  from  a "routine"  or  "harass"  posture  to  an  "attack" 
posture.  This  is  why  a system  which  assumes  a constant  data  generating 
process  does  not  respond  to  the  hostile  acts  as  rapidly  as  it  should.  We 
believe  that  a more  appropriate  model  might  be  a state-change  model.  The 
world  is  assumed  to  be  in  a particular  state  and  the  data  are  related  to 
the  question:  "Has  the  world  changed  from  its  initial  state  to  some  suc- 

ceeding state?"  The  initial  state  would  probably  be  routine  operations  in 
most  situations.  Each  datum  would  be  related  to  the  question:  "Has  the  world 

changed  from  routine  to  stage  or  routine  to  attack?"  Data  which  suggested 
routine  enemy  intent  would  simply  reassure  the  system  that,  if  a change  is 
going  to  take  place,  it  has  not  yet  occurred,  but  it  would  not  be  so  strongly 
diagnostic  of  a future  routine  intent  so  as  to  trap  the  bug. 

Unfortunately,  this  model  would  need  to  be  more  carefully  explored 
before  an  assessment  of  its  applicability  can  be  made.  We  offer  it  only  as 
a suggestion  as  a possible  starting  point  for  future  work. 

Responses  to  the  Questionnaire 

The  median  responses  to  the  scaled  items  of  the  questionnaire  are  given, 
along  with  the  questionnaire  items,  in  Appendix  B.  As  mentioned  above,  the 
major  purpose  of  the  questionnaire  was  to  monitor  the  conduct  of  the  investi- 
gation and  to  measure  the  effect  of  the  intended  "perceived  risk"  manipula- 
tion. However,  there  were  responses  to  some  items  which  are  informative 
and  germaine  to  the  discussion  of  the  decision  aids  themselves.  Most  notable 
was  the  strong  preference  of  the  subjects  for  the  DDI  triangle  over  the 
NAVPERSRANDCEN  bar  graph.  The  subjects  felt  that  the  bar  graph  contained 
more  information  but  found  the  triangle  display  easier  to  use.  By  their  own 


admission,  there  evidently  were  some  occasions  in  which  tiiey  were  confused 
by  the  bar  graph  display  and  basically  ignored  it.  This  adm iss ion  was  further 
supported  by  their  failure  to  strongly  disagree  witli  statement  number  7 on 
the  questionnaire.  We  feel  that  the  difference  in  ease  of  use  can  be  reduced 
or  even  eliminated  by  a little  additional  training  and  experience  with  the 
more  informative  bar  graph  display.  The  pressing  time  requirement  of  the 
subjects  prevented  us  from  being  more  thorough  in  training  them  in  the  use 
of  the  aids.  As  a consequence,  we  may  not  have  given  any  improvement  due 
to  explicit  risk  information  an  opportunity  to  manifest  itself. 


CONCLUSIONS 


The  results  indicate  that  the  ability  of  decision-aiding  systems,  such 
as  the  DDI  prototype,  to  aid  the  decision  maker  is  related  to  the  decision 
maker's  attitude  toward  risk.  The  performance  of  individuals  identified  as 
being  risk  averse  was  better  when  using  the  DDI  decision  aid.  Risk  pre- 
ferenced  individuals,  on  the  other  hand,  did  not  profit  from  the  use  of 
either  of  the  aids. 

Although  the  presence  of  information  in  addition  to  SEU  appeared  to 
be  of  some  use  to  the  risk  averse  group,  and  although  maximum  SEU  accounted 
for  less  than  75  percent  of  the  alternatives  that  were  selected,  the  exact 
nature  of  the  remaining  factors  of  choice  were  not  determined.  Both  minimum 
variance  and  minimax  strategies  were  ruled  out  as  major  factors  in  addition 
to  SEU. 

The  results  also  indicate  that  the  concept  of  estimating  likelihood 
ratios  for  anticipated  data  events  is  viable  as  long  as  there  is  an  opportunity 
for  the  decision  maker  to  revise  these  likelihoods  to  reflect  the  conditional 
dependencies  that  are  appropriate  for  the  occurrence  of  particular  data  com- 
binations. It  appears  the  estimation  of  canned  likelihood  ratios  might  best 
be  accomplished  by  some  group  estimation  procedure. 

The  expert  subjects  participating  in  this  research  felt  that  computer- 
based  decision  aids,  such  as  the  ones  used  in  this  study,  were  realistic 
goals.  The  subjects  particularly  liked  the  DDI  triangle  display  and  had 
no  difficulty  in  quickly  learning  to  use  it.  The  subjects  were  sometimes 
confused  by  the  more  complicated  bar  graph  display,  but  also  felt  that  it 
contained  useful  information  in  addition  to  that  which  they  received  from 
the  DDI  triangle. 


This  study  would  best  be  conducted  if  more  expert  subjects  were  avail- 
able and  tor  a longer  period  of  time.  One  hesitates  in  strongly  indicting 
or  strongly  supporting  the  approaches  discussed  here.  However,  no  clear 
cut  and  distinct  answer  can  ever  be  anticipated  in  simulation  research  such 
as  this.  The  purpose  of  this  study  is  served  by  its  failure  to  find  strong 
evidence  for  the  need  to  incorporate  certain  dimensions  of  risk  as  major 
components  of  an  operational  decision  aid.  In  the  process  of  the  investiga- 
tion, several  important  issues  were  uncovered  and  several  suggestions  for 
their  inclusion  were  determined. 
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RECOMMENDATIONS 


Based  upon  the  findings  of  this  investigation,  several  recommendations 
can  be  made  with  regard  to  the  further  development  and  application  of  decision 
aids  such  as  the  DDI  prototype. 

Insofar  as  the  dimensions  typically  defined  to  be  reflective  of  risk 
are  concerned,  there  is  no  strong  evidence  that  their  explicit  inclusion 
in  the  decision  aiding  display  is  particularly  beneficial.  This  is  not  to 
say  that  considerations  of  risk  are  unimportant,  but  that  the  study  was  not 
able  to  identify  any  particular  dimension  of  risk  that  was  important.  Unless 
subsequent  research  indicates  differently,  it  is  recommended  that  there  be 
continued  development  of  the  maximum  SEU  approach  employed  by  Decisions  and 
Designs,  Inc.  This  recommendation  is  based  primarily  upon  the  subjects' 
selection  of  the  maximum  SEU  act  in  almost  two-thirds  (62.1%)  of  the  trails — 
even  when  no  aid  was  present.  Although  the  presence  of  explicit  risk 
information  in  the  NAVPERSRANDCEN  version  did  not  enhance  performance 
significantly,  the  participants  in  the  investigation  felt  that  it  provided 
valuable  information  which  may  be  more  important  in  other  settings. 

Thus,  further  development  of  decision  analytic  aids  might  incorporate 
into  the  aids  the  ability  for  the  user  to  have  access  to  risk  information 
upon  request. 

The  aids  were  beneficial  only  to  those  subjects  who  became  more  risk 
averse  under  conditions  of  high  risk.  Thus,  the  use  of  such  aids  might  be 
limited  to  those  individuals  who  can  benefit  from  the  use  of  such  aids.  If 
so,  it  is  necessary  that  a reliable  technique  be  developed  to  make  that 
identification.  The  results  of  such  identification  could  impact  upon  training 
and/or  the  configuration  of  such  aids  available  to  the  decision  maker. 

If  a technique  to  be  used  for  decision  aiding  requires  reliable  subjective 
estimates,  as  does  the  Bayesian  approach,  some  consideration  should  be  given 
to  the  use  of  estimation  procedures  which  yield  increased  reliability.  One 
possibility  could  be  some  type  of  group  estimation  procedure. 

Additionally,  the  use  of  techniques  requiring  estimates,  such  as  the 
likelihood  ratio,  should  be  accompanied  by  the  ability  of  the  user  of  the 
decision  aid  to  update  or  change  the  estimate  to  reflect  conditional 
dependencies  in  the  data. 

There  is  also  a specific  recommendation  with  regard  to  the  Bayesian 
approach  used  by  DDI.  Because  of  the  distinct  problem  of  the  "trapped  bug," 
it  is  recommended  that  the  decision  task  associated  with  the  use  of  the  DDI 
aid  be  expanded  to  include  an  assessment  of  the  probability  that  the  enemy 
has  changed  its  intent.  This  is  in  addition  to  the  basic  task  of  diagnosing 
enemy  intent  and  selecting  the  appropriate  action. 
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SCENARIO  1 


I.  U.  S.  FORCES 

A.  CV  WITH  PLANE  GUARD  DESTROYER 

B.  DESTROYER  50  NM  AHEAD  OF  CV  (LEAD  DESTROYER) 

C.  DESTROYER  50  NM  BEHIND  CV  (TRAIL  DESTROYER) 

TASK  FORCE  IS  HEADING  090°  TOWARDS  ONRODA,  WHICH  LIES  1000  NM  AWAY. 

SOA  IS  12  KTS. 

II.  RED  FORCES 

A.  KOTLIN  TATTLETALE 

B.  KRESTA  I 100  NM  NORTH 

C.  KYNDA  70  NM  EAST 

ALL  BEARINGS  AND  DISTANCES  ARE  RELATIVE  TO  THE  CV  UNLESS  OTHERWISE  NOTED. 


KYNDA 


TIME  rtE  PREVIOUS  DATUM 


OE  CURRENT  DATUM 

0 12  APRIL 


U RUE  NT  UAI..U 

S-3  WORKING  S0N0BU0Y  PATTLRN 
90  NM  EAST  OF  THE  C V REPORTS 
INDICATIONS  OF  A CHARLIE 
SUBMARINE  CLOSING  ON  THE 
TASK  FORCE. 


1200 


KOTLIN 

KRESTA 


B 360  l 100  l 1200 


C I 090  I 70  I 1200 


OC5IG  | POSIT  TIME  BY  WmOm/mOw  rL*SS 

! — ■■■  j — ■ ■ —4 

A 090/90 ! 1 200  is- 3/BUOY  CHARLIE  iSS-N-7 


DATUM  CI.ASSI  El  CATION 

C.  SS-N-7  PLATFORM  ACTIVITY 


THREATENING  ACTION-WITHI 
WEAPONS  RANGE 
THREATENING  ACTION- 
OUTSIDE  WEAPONS  RANGE 
AMBIGUOUS  ACTION 
NON-THREATENING  ACTION 


1200  12  APRIL 


1237  12  APRIL 


— OBSERVED  FROM  KOTLIN 
TATTLETALE. 


KOTLIN 


KRESTA  I 


KVNDA 


DATUH  CLASSIFICATION 


F.  SURFACE-TO-SURFACE  GUN 
PLATFORM  ACTIVITY 


1.  THREATENING  ACTION 

2.  AMBIGUOUS  ACTION 

3.  NON-THREATENING  ACTION 


A S 

CONTACT  STATUS 

OCSIG 

POSIT 

Tl* 

BY  HHOM/KM 

CLASS 

Rf  MAMS 

A 

090/70 

1237 

S- 3/BUOY 

IHARLIE  (SS-N-7 





1237  12  APRIL 


1239  12  APRIL 


RV  KOTLIN  TATTLETALE 


CEASES 


KRESTA  I 


KYNDA 


S- 3/BUOY 


CHARLIE 


r.  SURFACE-TO-SURFACE  GUN 
PLATFORM  ACTIVITY 


1.  THRFATENING  ACTION 

2.  AMBIGUOUS  ACTION 

3.  NON-THREATENING  ACTION 


- 

•• 

| | 

•ill  Ml  M m ,t 

1 1 1 ! 



i | 

"T 1 " 

' — : — i 

A 

270 

1 ? 

1 1239  i 090  1? 

B 

360 

; 100 

1239  090  12 

C 

090 

70 

1 1239  090  12 

I _ i 

_L 

J 

1 1 I I 1 

_J ! 1 1 1 

12»  12  APRIL 


TIM  or  CUMWT  MTW 

1250  - 1254  12  APRIL 


CUUBVT  MTW 

BRIEF  PERI00  OF  INCREASED 
ENEMY  SURVEILLANCE  ACTIVITY 
OBSERVED. 


DATUM 

CLASSIFICATION 

M. 

ENEMY  SURVEILLANCE 

Liuma 

1. 

INCREASE 

2. 

NORMAL 

3. 

DECREASE 

1254  12  APRIL 


T1HZ  or  CVlUtNT  QATUH 

1530  12  APRIL 


CUAATKT  DATUM 

LEAD  DESTROYER  REPORTS 
FROM  BEARING  OF  KYNDA 


KRESTA  I 


KYNDA 


EAD  DESTR- 
IYER/SONAR 


HARLIE 


DATUM  CLASSIFICATION 

A.  SS-N-3  SURFACE  PLATFORM 
ACTIVITY 


THREATENING  ACTION-WITHIN 
WEAPONS  RANGE 
THREATENING  ACTION- 
OUTSIDE  WEAPONS  RANGE 
AMBIGUOUS  ACTION 
NON- THREATENING  ACTION 


otsir. 

POSIT  T I* 

A 

090/15 j 1530 

“ T | 

1 

E W 

STATUS 

_ J 

MAM 

TN 

• wo 

iHMt  Al 

Ml  MAw» 

.. 

_| 

- 



■ 

S* 

MM. 

NM 

1 INI 

- 

•,.v 

A 

270 

2 

1530 

090 

12 

B 

360 

100 

1530 

090 

12 

c 

090 

70 

1530 

090 

12 

“ 



TI*  OF  MllViOUS  DATUM 

1530  12  APRIL 


TI*  Of  CUDMMT  DATUM 

1531  12  APRIL 


FROM  BEARING  OF  KYNDA 


CEASES 


DATUM  CLASS  I FICATIOM 

A.  SS-N-3  SURFACE  PLATFORM 


THREATENING  ACTION-WITHIN 
WEAPONS  RANGE 
THREATENING  ACTION- 
OLITSIDE  WEAPONS  RANGE 
AMBIGUOUS  ACTION 
NON- THREATENING  ACTION 


PIATFOF9* 


_EAD  DESTR- 
)YER/ SONAR 

CHARLIE! 

SS-N-7 



1S31  12  APR  1 1 


Tint-  01  rilHHKN'r  DATUM 

1733  12  APRIL 


LEAD  DESTROYER  AGAIN 
REPORTS  FROM  THI 
BEARING  OE  KYNDA. 


;R!  STA  I 


LEAD  DESTR-! 
loYER/SONAR  tHARLIE 


090/ IS 


IIAH'M  iTASSlHlATlON 


A.  SS-N-3  SURFACE  PLATFORM 
ACTIVITY 


THRLATFNING  ACTI0N-W1TH 
WEAPONS  RANGE 
lHREATf  N1NG  ACT  ION- 
ODES  I Df  WEAPONS  RANG! 
AMBIGUOUS  ACTION 
NON- TMREATI  N I NO  ACl ION 


s u r r 

AM 

1 A ! II  S 

> 



II* 

A 

270 

2 

1733 

090  12 

II 

3M) 

100 

1/33 

090  12 

C 

090 



/() 



1731 

090  | 1 ? 

1 

1 

j 

• 

1736  12  APRIL 


FROM  SEARING  OF  KYNI 


CEASES 


KOTLIN 


KRESTA  I 


KYNOA 


i LEAD  OESTR- 

A 1090/15  |1736  I OYER/ SONAR  (CHARLIE 


DATUM  CLASS! riCATlOM 


SS-N-3  SURFACE  PLATFORM 


THREATENING  ACTION-HITHI 
WEAPONS  RANGE 
THREATENING  ACTION- 
OUTSIDE  WEAPONS  RANGE 
AMBIGUOUS  ACTION 
NON-THREATENING  ACTION 


«.«  1 M«C 

m»(Ca  I tine  I 

A ! 270 

1 

2 1736  , 090  ] 12 

8 360 

100  1736  J 090  12 

I | 

_C_  j.  Q9Q  . 

_J_ 

,__j [_ L 

T 

T1MF  OF  FREVIOUS  DATUM 


1736  12  APRIL 


TIME  OF  CURRENT  DATUM 

1755  12  APRIL 


CURAEMT  DATUM 

PLANE  GUARD  REPORTS  SONAR 
CONTACT  ON  PROBABLE  CHARLIE 
SUBMARINE  15  NM  NORTH  OF  THE 
CV. 


DATUM  CLASSIFICATION 

C.  SS-N-7  PLATFORM  ACTIVITY 

1.  THREATENING  ACTI0N-W1THIN 
WEAPONS  RANGE 

2.  THREATENING  ACTION- 
OUTSIDE  WEAPONS  RANGE 

3.  AMBIGUOUS  ACTION 

4.  NON- THREATENING  ACTION 


Tiw  or  previous  datum 

1755  12  APRIL 


1830-1834  12  APRIL 


CURREtrr  DATUM 

TASK  FORCE  AGAIN  DETECTS  A 
BRIEF  PERIOD  OF  ENEMY 
SURVEILLANCE  ACTIVITY 
INCREASE 


KRESTA  I 


KYNDA 


LEAD  DESTR- 

090/15  1834  OYER/SONAR  CHARLIE  SS-N-7 


PLANE 


DATUM  CLASS  I FI  CATION 

M.  ENEMY  SURVEILLANCE 
ACTIVITY 


360/15  1834  GUARO/SONARCHARL  It  SS-N-7 


T'.MI  or  .DEVIOUS  OATIW 


TINE  or  CUMINT  DATUM 

2145  12  APRIL 


CUSEEMT  DATUM 

P-3  WORKING  120  NM  WEST 
REPORTS  TRACKING  AN  I CHO  II 
SUBMARINE. 


DATUM  CLASSirifATlON 


B.  SS-N-3  SUBMARINE  PLAT 
FORM  ACTIVITY 


THREATENING  ACTION  WITHIN 

WEAPONS  RANGE 

THRI  ATI  NING  AC  I I ON 

OUT 5 1 01  WEAPONS  HANOI 

AMBIGUOUS  ACTION 

NON- THRI  ATI  NING  AC  I ION 


L ’ 

... 

** 

- ; - 

... 

-- 

SURF 

A C f 

ST*  ' US 

J 

• 

• J 

A 270  2 

.’I4‘> 

090 

1? 

*■  t 

B 360  100 

P14‘. 

090  I 

1? 

| ] 

j * 

C 000  j /() 

/I4S 

0*10 

12 

-f  — 

_1. 

[ 

E w 

STATU 

* ' ' ' 1 

RA. 

>« 

iRIV 

». 

* *M| 

• ^ 

- 

— 

- ! 

ilC  s i 

RQS  IT 

Y .Mf 

ASS 

.A.J 

090/15 

2145 

LEAD  OE STR- 
OVE R/ SONAR 

CHARLIE  SS-N- 7 

B 

360/ IS 

2145 

FTRUT 

GUARO/ SONAR 

CHARI  II'  SS-N- 7 

C 

270/120 

2145 

3/ BUOY 

ICHO  II  SS-N-3 

KRESTA  I 


nm  or  cuuiirT  datum 

0100  13  APRIL 


DATUM  CLASSIFICATION 

G.  TARGETING  PLATFORM  (BEAR 


1. 

2. 

3. 


nm  or  ritvious  datum 

2145  12  APRIL 


CUUtVr  DATUM 


RADAR  CONTACT  ON  TARGET 
INBOUND  FROM  030°/—  NM  AT 
30000' , SPEED  250  KTS. 
TENTATIVELY  CLASSIFIED  AS  A 
BEAR  D. 


THREATENING  ACTION-NO  BIG 
BULGE  OR  VDL 
THREATENING  ACTION-BIG 
BULGE  BUT  NO  VDL 
THREATENING  ACTION-VDL 
AMBIGUOUS  ACTION 
NON- THREATENING  ACTION 


300  ONE  ,0100  BEAR  D 


| LEAD  DESTR-( 

A 090/15  0100  | OYER/SONAR  CHARLIE jSS-N-7 

[PLANE 

B | 360/15  I 0100  GUARD/ SONAljCHARL I E iSS-N-7 

C j 270/ 120l  0100  P- 3/BU0Y  ECHO  II  SS-N-3 


TJ  ME  OF  EREVJOUS  DATUM 

0100  13  APRIL 


0112  13  APRIL 


CURRENT  DATUM 

— OBSERVED  FROM  THE  INBOUND 
BEAR. 


KYNDA 


DATUM  CLASSIFICATION 

G.  TARGETING  PLATFORM  (BEAR 
0/ HORMONE)  ACT  I VI T Y 


THREATENING  ACTION-NO  BIG 
BULGE  OR  VOL 
THREATENING  ACTION-BIG 
BULGE  BUT  NO  VDI 
THREATENING  ACTION- VDI 
AMBIGUOUS  Ad  ION 
NON-THRI  All  NINO  ACTION 


A S 

CONI 

act  s i a t 

IV), II 

' !•*< 

.. 

LEAD  OESTR- 

A 

090/15 

0112 

OYER/SONAR  CHARLIE  SS-N-7 

PLANE 

B 

360/15 

0112 

GUARD/SONAR  CHARLIE  SS-N-7 

C 

270/120 

0112 

P- 3/BUOY  ECHO  I I SS-N-3 

K 0 I,  1 Y 1 M • l 

» 

II. 

030 

1 

210  ' 

- | TOO  , ONI  I 0 1 1 2 ' ISIAR  0 

- - i 1 

I 

| 1 ' : 

—f 

i 

. i 

1 1 • . . 

tiw  or  ClHUtEKT  DATUM 

0137  13  APRIL 


CUUtENT  DATUM 

BEAR  AIRCRAFT  BEGINS 
ORBITING  AT  — NM. 


DATUM  CLASSIFICATION 


G.  TARGETING  PLATFORM  (BEAR 
D/HORMONE)  ACTIVITY 


THREATENING  ACTION-NO  BI 
BULGE  OR  VDL 
THREATENING  ACTION-BIG 
BULGE  BUT  NO  VDL 
THREATENING  ACTION-VDL 
AMBIGUOUS  ACTION 
NON- THREATENING  ACTION 


A $ 1 

ACT  r ’ « 

l«M< 

LEAD  DFSTR- 

A 

090/15  0137 

OYER/SONAR 

CHARLIE  SS-N-7 

PLANE 

B 

360/15  0137 

GUARD/ SONAR 

CHARLIE  SS-N-7 

C 

270/120  0137 

P-3/BU0Y 

ECHO  II'  SS-N-3 

U R r A C f 

A T J 

c 

’ 

Ai 

270 

2 i 0137 

090 

12 

B 

360 

100  0137 

09C 

12 

C 

090 

70  0137 

090 

12 

i 

_ . J 

0137  13  APRIL 


TIME  or  CURKENT  DATUM 

0150  13  APRIL 


OBSERVED  FROM  THE 
ORBITING  BEAR. 


I I AO  111  STR- 

lOYIR/SONAK  lHAKI  II  55-5  1 
PI  ANI 

GUARD/ SONAR  CHARI  II  SS-N-/ 


1(190/15  0150 


l*A  IHM  < I*  . .1  M i ATHfi 

G.  IARGLTING  PLATFORM  (BEAR 
0/ hormone)  Activity 


R 360/15  I 0150 


70/1 20  0150 


P- 3/BUOY  ICHO  II'  SS-N- 3 


THREATENING  AC1 ION-NO  BIG 
BULGE  OR  VDl 
THREATENING  AH  ION-BIG 
Bill  Gf  BUT  NO  VOl 
THRLATINING  AT  1 I (IN- VOl 
AMBIGUOUS  ACTION 
NON- THREATENING  ACTION 


SURF 

AfJ  ! A ' II  \ 

• 

••A, 

A 

270  | 2 

0150  090  IP 

B 

360  , 100 

0150  090  1.’ 

C 

090  ' 70 

\ 

0150  090  1/ 

LEAD  DESTR- 
OYER/ SONAR  CHARLIE 


090/15  0152 


PLANE 


GUARD/SONAf}  CHARLIE 


360/15  S 0152 


P-3/BU0Y 


270/120!  0150 


TtHI  OF  CUUCHT  DATUM 

0152  13  APRIL 


CUUEMT  DATUM 

P-3  REPORTS  A SURFACE  CONTAC 
IN  THE  AREA  WHERE  IT  HAS  BEEI 
TRACKING  THE  ECHO  II  SUB- 
MARINE. 


n*  or  ramous  datui 
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TROM  BEARING  OF  KYNDA 
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KYNDA 
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DATUM  CLASSIFICATION 

K.  LAUNCHED  MISSILE  (SS) 


HIGH  SPEED  (1.3  MACH) 
TARGET  DETECTED  INBOUND  TO 
THE  CV  FROM  360°/10NM  AT 
700'.  CLASSIFIED  AS  A 
VAMPIRE. 


TI«  OF  FMVIOUS  DATUM 


0154  13  APRIL 


TIMS  OF  C'JMZNT  DATUM 


0154  13  APRIL 
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C 1270/120  0150  P- 3/BUOY  ECHO  II  SS-N-3 


DEBRIEFING  QUESTIONNAIRE 


I.  Please  use  the  following  scale  to  indicate  how  much  you  agree  with  each 
of  the  following  statements. 


Strongly 

Disagree 


Strongly 

Agree 


MdN 

Response 

4.5 

1. 

4.75 

2. 

4.16 

3. 

2.00 

4. 

3.83 

5. 

2.00 

6. 

2.10 

7. 

2.30 

8. 

1.83 

_ 9. 

4.25 

10. 

2.00 

11. 

4.50 

12. 

3.30 

13. 

3.70 

14. 

1 2 3 4 5 

The  use  of  computer  based  systems  for  decision  aiding  in 
tactical  operations  is  a realistic  goal  for  the  near  future. 

Considering  the  limited  time  available,  the  briefing  and 
instructions  were  sufficient  for  using  the  decision  aids. 

The  problems  used  in  the  study  were  realistic  and  plausible. 

When  there  were  3 CVA's  available  instead  of  1 CVA,  it  was 
more  important  to  select  the  correct  action. 

It  was  relatively  easy  to  classify  each  data  event  as  belong- 
ing to  one  of  the  categories  given  at  the  bottom  of  each  page. 

There  were  many  occasions  in  which  the  appropriate  classification 
was  not  one  of  the  alternatives  available. 

I didn't  pay  much  attention  to  the  information  presented  in 
the  display  . 

I sometimes  disagreed  with  the  display  at  first,  but  noticed 
information  that  convinced  me  that  I was  wrong. 

None  of  the  displays  seem  to  be  particularly  helpful  in 
analyzing  the  tactical  situations. 

The  triangle  display  was  more  useful  than  the  bar  graph  display. 
The  bar  graph  display  was  more  useful  than  the  triangle  display. 
The  triangle  display  was  the  easiest  to  use. 

The  bar  graph  contained  more  information  than  the  triangle  dis- 
play. 

The  concept  of  "likelihood"  wa3  easy  to  comprehend. 


B-l 


II.  The  following  questions  will  only  be  used  for  evaluating  the  utility  of 
the  various  aids  relative  to  your  experience. 

A.  Military  rank?  

B.  Number  of  years  in  the  Navy? 

C.  What  were  some  of  the  difficulties  you  had  with  the  displays? 


D.  What  pieces  of  information  would  enhance  the  use  of  the  aid  for 
decision  making? 


E.  Which  aid  was  the  most  useful,  and  why  did  you  prefer  it? 


III.  Lf  a "follow-up"  study  is  conducted  in  the  near  future,  would  you  be 
willing  to  participate  for  a couple  of  hours  at  your  convenience? 


B-2 


PARTICIPANT  if 


ASSESSMENT  OF  RED  INTENT  INDICATORS 
INDICATOR  RED  INTENT 

ROUTINE  HARASS  ATTACK 

A.  SS-N-3  Surface  Platform  Activity: 

1.  Threatening  action-within  weapons 
range 

2.  Threatening  action-outside  weapons 
range 

3.  Ambiguous  action 

4.  Non-threatening  action 

B.  SS-N-3  Submarine  Platform  Activity: 

1.  Threatening  action-within  weapons 
range 

2.  Threatening  action-outside  weapons 
range 

3.  Ambiguous  action 

4.  Non-threatening  action 

C.  SS-N-7  Platform  Activity: 

1.  Threatening  action-within  weapons 
range 

2.  Threatening  action-outside  weapons 
range 

3.  Ambiguous  action 

4.  Non-threatening  action 

D.  SS-N-2/SS-N-11  Platform  Activity: 

1.  Threatening  action-within  weapons 
range 

2.  Threatening  action-outside  weapons 
range 

3.  Ambiguous  action 

4.  Non-threatening  action 

C-l 


PARTICIPANT  # 


ASSESSMENT  OF  RED  INTENT  INDICATORS 
INDICATOR  RED  INTENT 

ROUTINE  HARASS  ATTACK 

E.  SAM  Platform  Activity: 

1.  Threatening  action 

2.  Ambiguous  action 

3.  Non-threatenting 

F.  Surface-to-Surface  Gun  Platform 
Activity: 

1.  Threatening  action 

2.  Ambiguous  action 

3.  Non-threatening  action 

G.  Targeting  Platform  (Bear  D/Hormone) 

Activity : 

1.  Threatening  action-no  Big  Bulge  or 
VDL 

2.  Threatening  action-Big  Bulge  but 
no  VDL 

3.  Threatening  action-VDL 

4.  Ambiguous  action 

5.  Non-threatening  action 

H.  AS-1/2/3  Etc  Platform  Activity: 

1.  Threatening  action-within  weapons 
range 

2.  Threatening  action-outside  weapons 
range 

3.  Amblgous  action 

4.  Non-threatening  action 


C-2 


PARTICIPANT 


ASSESSMENT  OF  RED  INTENT  INDICATORS 


INDICATOR 

I.  Tactical  Aircraft  Activity: 

1.  Threatening  action 

2.  Ambiguous  action 

3.  Non-threatening  action 

J.  Other  Aircraft  (e.g.  Badger  D/E) 
Activity: 

1.  Threatening  action 

2.  Ambiguous  action 

3.  Non-threatening  action 

K.  Launched  Missile  fSS)  Indication: 

1.  Strong  indicator 

2.  Weak  indicator 

L.  Launched  Missile  (SA)  Indication: 

1.  Strong  indicator 

2.  Weak  indicator 

M.  Enemy  Surveillance  Activity: 

1.  Increase 

2.  Normal 

3.  Decrease 

N.  Enemy  Communication  Activity: 


1. 

Increase 

2. 

Normal 

3. 

Decrease 

ROUTINE 


RED  INTENT 


HARASS 


1 


ATTACK 
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THE  DETERMINATION  OF  SUBJECT  PERFORMANCE  SCORES 


The  scenario  author  had  designed  some  scenarios  as  ultimately  displaying 
an  overall  Red  intent  of  conducting  either  routine  operations  or  harassment. 
For  those  scenarios,  the  subject  received,  as  a score  at  each  page  of  the 
scenario,  that  value  from  his  appropriate  utility  table  corresponding  to 
the  action  he  selected  at  that  page  and  the  Red  intent  which  one  might 
assume  to  exist  (according  to  the  scenario  author),  based  upon  all  available 
information  up  to  that  point.  A subject's  overall  score  for  a routine  or 
harass  scenario  was  then  obtained  by  averaging  scores  for  individual  pages. 
However,  if,  at  any  point  in  a routine  or  harass  scenario,  a subject  had 
decided  to  attack,  then  his  scenario  score  was  the  value  from  his  appropriate 
utility  table  corresponding  to  an  attack  action  and  a Red  intent  of  routine, 
for  a routine  scenario,  or  of  harassment,  for  a harass  scenario. 

For  attack  scenarios,  the  scenario  author  identified  the  approximate 
time,  in  his  opinion,  the  first  Red  weapon  could  have  struck  the  Blue  force, 
given  the  unfolding  scenario  events.  He  also  identified  two  additional 
times,  the  "attack-late"  and  "stage-late"  times.  The  attack-late  time  was 
the  latest  time  Blue  could  have  launched  an  anticipatory  attack  which  would 
have  gotten  weapons  on  all  Red  targets  prior  to  Red  launching  weapons.  The 
stage-late  time  was  the  latest  time,  in  the  scenario  writer's  opinion,  that 
Blue  could  have  commenced  taking  appropriate  defensive  measures  (e.g.,  general 
quarters  and  launching  alert  aircraft)  to  reach  a state  of  full  defensive 
readiness  by  Red  weapon  impact  time.  To  avoid  biasing,  all  these  times  were 
identified  by  the  scenario  author  before  any  data  inspection  occurred. 

Utilizing  these  times,  a subject's  score  for  an  attack  scenario  was 
determined  as  follows.  If,  at  any  point  in  the  scenario  prior  to  the  attack- 
late  time,  the  subject  decided  to  attack,  his  score  for  that  scenario  was 
the  value  from  his  appropriate  utility  matrix  corresponding  to  the  Blue 
attack/Red  attack  combination.  However,  if  he  chose  to  attack  after  the 
attack-late  time,  his  scenario  score  was  a linear  interpolation,  based  upon 
the  tardiness  of  his  attack  decision,  between  his  utilities  for  the  Blue 
attack/Red  attack  combination  and,  depending  upon  his  action  decision  just 
prior  to  the  attack  decision,  the  Blue  stage/Red  attack  combination  or  the 
Blue  routine/Red  attack  combination.  If  a subject  never  made  an  attack 
decision  in  an  attack  scenario,  his  score  was  determined  in  a similar  manner 
by  utilizing  the  "stage"  row  of  his  utility  matrix  and  the  stage-late  time. 
There  were  no  cases  where  a subject  decided  to  maintain  routine  operations 
throughout  an  attack  scenario.  Since  the  values  from  the  utility  matrix 
were  the  subject's  own  estimates  of  the  cost  of  various  outcome,  and  because 
these  utility  scores  were  adjusted  for  timeliness,  an  approximation  of  the 
actual  outcome  of  the  scenario  could  be  calculated  even  though  the  scenario 
was  not  continued  to  its  conclusion. 
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